Could we perhaps summarize (beyond the revisions I already agreed to make in my Word-file replies) what I must do next?
To be clear: I approve the paper, conditioned on the agreed up revisions.
Back to [Archive] Post-review discussions
To be clear: I approve the paper, conditioned on the agreed up revisions.
REPLY: Would it be ok to include “(but, see, <your link>)” immediately after the word “version” in our quote above? If not, we can delete this entire section.
No. Please see the discussion about your previous submission. The BDS does not have twice the g-loading of FDS. Dalliard and myself compiled several studies to show this, yet you made the same exact claim in your next submission. I find that odd.
That section shall be removed from the revision.
This (the slightly higher mean IQ among the occupations) corresponds to the low correlation found between being out of a job and IQ at the individual level. You could back-estimate this correlation using this mean. Just a minor check.
See my reply to Chuck re my thoughts on additional analyses like these. I will defer, if that's what consensus here demands.
You misunderstood. I proposed that you use the average of the BLS 2014 and 2012 values to remove some of the slight 'measurement error'.
I did this in my replication.
I admit to not being a statistician, but to me, I found it more compelling to show that the demographic percentages were stable across two time periods than to get slightly more precise estimates by averaging them.
1)
Not controlling for known confounds (interests in this case) which you already have the data for is not really defensible. After all, you are interested in the effect of cognitive ability itself, not whatever it is that it happens to be correlated with. If you use correlations, you will get a confounded estimate of the influence of cognitive ability itself.
I misinterpreted what you meant last time, and see that you have done the analyses below. How shall I proceed?
Your correlations are effect sizes, yes. However, I asked for "the effect sizes of the prior research so readers can see whether the effect sizes are similar".
You present some new results. What the readers need to know is whether they fit in size with the previous results. For instance, if you find r = .20 and previous studies have found r = .95, something is wrong somewhere.
The data variable (complexity) was correlated with mean IQ at .86 in your study. You cite:
Gottfredson, L. S. (1986). Occupational aptitude patterns map: Development and implications for a theory of job aptitude requirements (Monograph). Journal of Vocational Behavior, 29, 254-291.
Gottfredson, L. S. (2003). g, jobs, and life. In H. Nyborg (Ed.), The scientific study of general intelligence: Tribute to Arthur R. Jensen (pp. 293-342). New York: Pergamon.
However, I could not find any complexity x mean IQ correlation in these papers. She does give job mean IQs and presents factor analysis results of job attributes, but does not appear to actually correlate them. Maybe I missed the number somewhere?
I will look into this further. It’s possible Gottfredson didn’t report any.
Bryan,
Some comments.
There may be some misunderstanding here. Sorry if I have not been entirely clear.
My point is that:
1) There is year-to-year random variation in the BLS race% data.
2) By correlating data from different years, one can see how large this variation is. You did this and found that there is little variation (r's = .87 to .88).
3) By averaging data across years, the random variation cancels out in accordance with the Spearman-Brown formula. This should improve the true score variance (i.e. the signal in the noise).
So, I'd like you to use the average values of years 2012 and 2014 for the BLS data, and note their intercorrelation (as you already do).
I think that your correlations are good to include. They are the simplest test one can do of this hypothesis with these data.
Furthermore, I think that you should include the regression results. I.e. have each of the race% as outcome (dependent) variables, and use mean IQ, people, and things as predictors (independents). This gives a less confounded view on the effect of GCA alone (which was in fact pretty stable across racial groups).
I don't think the path model results are interesting enough to add.
Please do. However, it might simply mean that your study is more novel than you thought.
Perhaps contact Linda Gottfredson to hear. Perhaps she is familiar with other literature on this topic.
---
As Fuerst notes, it is a good idea to note that the White, Black, Asian groups in this study include Hispanics. Hispanics mostly self-identify as White or Other, so these are the groups that are 'contaminated' and harder to interpret results for.
https://en.wikipedia.org/wiki/Hispanic_and_Latino_Americans#Race
The BLS report states (p. 59):
---
One should also do the analyses for Hispanic%. It is not important, statistically, that this group overlaps with the others because you are not doing a simultaneous analyses of the race%'s.
Some comments.
I admit to not being a statistician, but to me, I found it more compelling to show that the demographic percentages were stable across two time periods than to get slightly more precise estimates by averaging them.
There may be some misunderstanding here. Sorry if I have not been entirely clear.
My point is that:
1) There is year-to-year random variation in the BLS race% data.
2) By correlating data from different years, one can see how large this variation is. You did this and found that there is little variation (r's = .87 to .88).
3) By averaging data across years, the random variation cancels out in accordance with the Spearman-Brown formula. This should improve the true score variance (i.e. the signal in the noise).
So, I'd like you to use the average values of years 2012 and 2014 for the BLS data, and note their intercorrelation (as you already do).
I misinterpreted what you meant last time, and see that you have done the analyses below. How shall I proceed?
I think that your correlations are good to include. They are the simplest test one can do of this hypothesis with these data.
Furthermore, I think that you should include the regression results. I.e. have each of the race% as outcome (dependent) variables, and use mean IQ, people, and things as predictors (independents). This gives a less confounded view on the effect of GCA alone (which was in fact pretty stable across racial groups).
I don't think the path model results are interesting enough to add.
I will look into this further. It’s possible Gottfredson didn’t report any.
Please do. However, it might simply mean that your study is more novel than you thought.
Perhaps contact Linda Gottfredson to hear. Perhaps she is familiar with other literature on this topic.
---
As Fuerst notes, it is a good idea to note that the White, Black, Asian groups in this study include Hispanics. Hispanics mostly self-identify as White or Other, so these are the groups that are 'contaminated' and harder to interpret results for.
https://en.wikipedia.org/wiki/Hispanic_and_Latino_Americans#Race
The BLS report states (p. 59):
White, Black or African American, Asian, American Indian and Alaska Native, and Native Hawaiian and Other Pacific Islander. In accordance with the Office of Management and Budget guidelines, these terms are used to describe the race of people. Beginning in 2003, people in these categories are those who selected that race group only. Those who identify multiple race groups are categorized as people of Two or More Races. (Previously, people identified a group as their main race.) People who identified themselves as Asian are further classified as Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, or Other Asian. The Other Asian category includes individuals of group not listed—such as Pakistani, Hmong, and Cambodian— and those who reported two or more Asian groups. Estimates for American Indians and Alaska Natives, Native Hawaiians and Other Pacific Islanders, and people of Two or More Races are not shown separately in all tables because the number of survey respondents is too small to develop estimates of sufficient quality. In the enumeration process, race is determined by the household respondent. More information on the 2003 changes to questions on race and Hispanic ethnicity is available on the BLS website at www.bls.gov/cps/rvcps03.pdf.
Hispanic or Latino ethnicity. This refers to people who identified themselves in the enumeration process as being of Hispanic, Latino or Spanish origin. These individuals are further classified by detailed Hispanic ethnicity. Previous versions of this report presented data for the following detailed Hispanic ethnicity categories: Mexican, Puerto Rican, Cuban, Central and South American, or Other Hispanic or Latino. The latter two categories were expanded in 2014 into additional categories: Central American, which includes the two subcategories of Salvadoran and Other Central American (excluding Salvadorans); South American; and Other Hispanic or Latino, which includes the two subcategories of Dominican and Other Hispanic or Latino (excluding Dominicans). People whose ethnicity is identified as Hispanic or Latino may be of any race. More information on the 2003 changes in questions on race and Hispanic ethnicity is available online at www.bls.gov/cps/rvcps03.pdf
---
One should also do the analyses for Hispanic%. It is not important, statistically, that this group overlaps with the others because you are not doing a simultaneous analyses of the race%'s.
Bryan,
So, I'd like you to use the average values of years 2012 and 2014 for the BLS data, and note their intercorrelation (as you already do).
...
Furthermore, I think that you should include the regression results. I.e. have each of the race% as outcome (dependent) variables, and use mean IQ, people, and things as predictors (independents).
....
Perhaps contact Linda Gottfredson to hear. Perhaps she is familiar with other literature on this topic.
....
As Fuerst notes, it is a good idea to note that the White, Black, Asian groups in this study include Hispanics.
....
One should also do the analyses for Hispanic%. It is not important, statistically, that this group overlaps with the others because you are not doing a simultaneous analyses of the race%'s.
Emil,
For sake of transparency -- I've asked this of other reviewers, too -- could you clarify which changes you absolutely require for approval versus which you recommend but do not insist upon? Thanks.
Bryan,
So, I'd like you to use the average values of years 2012 and 2014 for the BLS data, and note their intercorrelation (as you already do).
...
Furthermore, I think that you should include the regression results. I.e. have each of the race% as outcome (dependent) variables, and use mean IQ, people, and things as predictors (independents).
....
Perhaps contact Linda Gottfredson to hear. Perhaps she is familiar with other literature on this topic.
....
As Fuerst notes, it is a good idea to note that the White, Black, Asian groups in this study include Hispanics.
....
One should also do the analyses for Hispanic%. It is not important, statistically, that this group overlaps with the others because you are not doing a simultaneous analyses of the race%'s.
Emil,
For sake of transparency -- I've asked this of other reviewers, too -- could you clarify which changes you absolutely require for approval versus which you recommend but do not insist upon? Thanks.
Thanks for the query, Chuck. In the spirit of this, I'm ok with doing all of Emil's analysis, except including Hispanics.
Of my comments, I require:
1. The regression analyses to control for the interest differences.
I recommend but do not require:
1. Using the average of BLS 2012 and BLS 2014 values.
2. Doing the Hispanic analyses.
3. Contact Linda Gottfredson to hear if someone else correlated job complexity with mean IQs by job/occupation.
After (1), I have no further objections and will approve of publication.
---
I'm curious as to why you don't want to do the Hispanic one? In fact, the Hispanic one is the only one that uses a clear definition! It's the White, Black and Asian which are confounded with Hispanics (mostly White).
1. The regression analyses to control for the interest differences.
I recommend but do not require:
1. Using the average of BLS 2012 and BLS 2014 values.
2. Doing the Hispanic analyses.
3. Contact Linda Gottfredson to hear if someone else correlated job complexity with mean IQs by job/occupation.
After (1), I have no further objections and will approve of publication.
---
I'm curious as to why you don't want to do the Hispanic one? In fact, the Hispanic one is the only one that uses a clear definition! It's the White, Black and Asian which are confounded with Hispanics (mostly White).
Of my comments, I require:
1. The regression analyses to control for the interest differences.
I recommend but do not require:
1. Using the average of BLS 2012 and BLS 2014 values.
2. Doing the Hispanic analyses.
3. Contact Linda Gottfredson to hear if someone else correlated job complexity with mean IQs by job/occupation.
After (1), I have no further objections and will approve of publication.
---
I'm curious as to why you don't want to do the Hispanic one? In fact, the Hispanic one is the only one that uses a clear definition! It's the White, Black and Asian which are confounded with Hispanics (mostly White).
Hello,
I will do the regressions and all other changes suggested earlier. I just can't get past checking a box that says white, e.g., and then also a box that says Hispanic. That plus things summing to greater than 100% concerns me.
Of my comments, I require:
1. The regression analyses to control for the interest differences.
I recommend but do not require:
1. Using the average of BLS 2012 and BLS 2014 values.
2. Doing the Hispanic analyses.
3. Contact Linda Gottfredson to hear if someone else correlated job complexity with mean IQs by job/occupation.
After (1), I have no further objections and will approve of publication.
---
I'm curious as to why you don't want to do the Hispanic one? In fact, the Hispanic one is the only one that uses a clear definition! It's the White, Black and Asian which are confounded with Hispanics (mostly White).
Hello,
I will do the regressions and all other changes suggested earlier. I just can't get past checking a box that says white, e.g., and then also a box that says Hispanic. That plus things summing to greater than 100% concerns me.
Also, may I ask why you didn't include Data in with the regressions you want me to report?
Hi Bryan,
It is of no statistical importance that the groups do not sum to 100% because you do not use multiple groups at the same time. In fact, one could create pseudo-groups if one wanted to. As long as they have a known mean IQ, one can do this type of analysis. For instance, one could create a group of persons who are all descended from at least one parent with a college degree. Then another group with two parents with college degrees, and another group with 0 parents with college degrees. These three groups sum to more than 100%, but they would work fine for doing this analysis.
Of course, these would not be racial groups. But then again, there are other things to research than race. :)
Data is more or less the same as mean IQ requirement (rated job complexity). However, since we have the actual mean IQ by occupation, using both would create strong multicollinearity which causes problems using OLS regression. Here's an example:
With IQ:
With data:
With both:
So, using both does not improve the predictive validity. Cross-validation R2 is .19 for the model with both, and .19 and .21 for the other two (trivial difference). In this case we can see that what happens is that data (job complexity) takes the validity from IQ. This happens because it just so happens that the relationship with data is slightly stronger (.44 vs. .39). It might as well have been the other away around (the sample size is small). When two predictors are highly correlated, results from OLS regression are unstable and have large standard errors.
You can see the standard error inflation in this case. When the model includes only one of the predictors, the standard error is .09. When they are both included, the standard error increases for both by about 67% to .15.
I will do the regressions and all other changes suggested earlier. I just can't get past checking a box that says white, e.g., and then also a box that says Hispanic. That plus things summing to greater than 100% concerns me.
It is of no statistical importance that the groups do not sum to 100% because you do not use multiple groups at the same time. In fact, one could create pseudo-groups if one wanted to. As long as they have a known mean IQ, one can do this type of analysis. For instance, one could create a group of persons who are all descended from at least one parent with a college degree. Then another group with two parents with college degrees, and another group with 0 parents with college degrees. These three groups sum to more than 100%, but they would work fine for doing this analysis.
Of course, these would not be racial groups. But then again, there are other things to research than race. :)
Also, may I ask why you didn't include Data in with the regressions you want me to report?
Data is more or less the same as mean IQ requirement (rated job complexity). However, since we have the actual mean IQ by occupation, using both would create strong multicollinearity which causes problems using OLS regression. Here's an example:
With IQ:
> lm_white = lm("white ~ iq + people + things", data = d_jobdata) %>% MOD_summary(runs = 200)
> lm_white
$coefs
Beta SE CI.lower CI.upper
iq 0.39 0.09 0.20 0.57
people -0.19 0.10 -0.37 0.00
things -0.09 0.08 -0.26 0.07
$meta
N R2 R2 adj. R2 10-fold cv
124.00 0.26 0.24 0.19
With data:
> lm_white = lm("white ~ data + people + things", data = d_jobdata) %>% MOD_summary(runs = 200)
> lm_white
$coefs
Beta SE CI.lower CI.upper
data -0.44 0.09 -0.62 -0.27
people -0.16 0.09 -0.34 0.02
things -0.08 0.08 -0.24 0.08
$meta
N R2 R2 adj. R2 10-fold cv
124.00 0.29 0.28 0.21
With both:
> lm_white = lm("white ~ iq + data + people + things", data = d_jobdata) %>% MOD_summary(runs = 200)
> lm_white
$coefs
Beta SE CI.lower CI.upper
iq 0.07 0.15 -0.23 0.37
data -0.39 0.15 -0.69 -0.09
people -0.15 0.09 -0.34 0.03
things -0.08 0.08 -0.24 0.08
$meta
N R2 R2 adj. R2 10-fold cv
124.00 0.30 0.27 0.19
So, using both does not improve the predictive validity. Cross-validation R2 is .19 for the model with both, and .19 and .21 for the other two (trivial difference). In this case we can see that what happens is that data (job complexity) takes the validity from IQ. This happens because it just so happens that the relationship with data is slightly stronger (.44 vs. .39). It might as well have been the other away around (the sample size is small). When two predictors are highly correlated, results from OLS regression are unstable and have large standard errors.
You can see the standard error inflation in this case. When the model includes only one of the predictors, the standard error is .09. When they are both included, the standard error increases for both by about 67% to .15.
Hello,
I think we addressed all mandated concerns, and I did try to proofread carefully. If anyone finds a typo, please let me know.
Thanks for considering our work,
Bryan
I think we addressed all mandated concerns, and I did try to proofread carefully. If anyone finds a typo, please let me know.
Thanks for considering our work,
Bryan
Any update?
Bryan
Bryan
I have more comments, especially with regards to the theoretical framing. However, I've been too busy to devote time to this. I should have more time after the 1st July.
Any update?
Bryan
I will review it. Please let me some time. Thanks. (I have been busy with my own article, among other things...)
Jensen's method (correlated vectors)
When one has multiple, preferably many, indicators of a latent trait one can use Jensen's method to check whether this latent variable is related to a criterion variable or if its the other variance components (group factors or test/item specificity). This method requires that one has multiple indicators each of which have some measurement of how well they measure the latent trait. Usually factor loadings from factor analysis are used for this purpose, but if one has item-level data, one should use item response theory-based discrimination values instead.
It is true that jobs are basically mental tests of varying difficulty. SH for this kind of data would be the claim that the jobs where the g-job performance link is stronger would show larger group differences in job performance, assuming no other effects (such as selective hiring which is ubiquitous). However, hat is not the kind of data this study has. The data here are the racial proportions of each job and some information about the jobs. One cannot frame the current study as a test of SH.
However, I still think the present study is useful and I have no serious criticism of the methods used, but it's a study of something else. It's a study of what happens when one has groups with different mean ability levels and there are jobs that select for different levels of cognitive ability. In the absence of differential hiring, there should be more of the higher scoring groups in the jobs that recruit from higher up the cognitive ability distribution. Of course, we know that there is some differential hiring and using IQ tests without doing this may be illegal (but not in the military).
Preferably, one should try to model the racial proportions together based on demographic data and assumptions about the range each job recruits workers from. However, the authors prefer to take a simpler approach and check the simpler prediction that the jobs with higher means do have more persons from the higher ability groups. This is fine with me.
Write-up of regressions
The write-up of the additional regression analyses is not satisfactory. These regressions were not based on any pre-analysis hypothesizing as the writing says. They were entirely exploratory, post hoc regressions and should be clearly labeled as such. To do otherwise, is to HARK (http://psr.sagepub.com/content/2/3/196.abstract). I would write something like:
A reviewer* suggested using multiple regression by including the jobs' ratings for whether they involve working with people and things. This was done because there may be group differences in these preferences which may obscure the relationship to the mean levels of cognitive ability. A regression model was fit for each race% as the outcome and with the mean job IQ, people-interest rating and person-interest rating as the predictors. [3 tables of results] The mean IQ was a good predictor in all the models and, importantly, the small correlation seen for Asian% seemed to be due to a suppression effect from a confound with a relatively stronger preference among Asians for working with people.
I'm sorry, but the framing of the study has to be changed. This is not a test of Spearman's hypothesis.
* I usually name reviewers who give useful suggestions.
When one has multiple, preferably many, indicators of a latent trait one can use Jensen's method to check whether this latent variable is related to a criterion variable or if its the other variance components (group factors or test/item specificity). This method requires that one has multiple indicators each of which have some measurement of how well they measure the latent trait. Usually factor loadings from factor analysis are used for this purpose, but if one has item-level data, one should use item response theory-based discrimination values instead.
It is true that jobs are basically mental tests of varying difficulty. SH for this kind of data would be the claim that the jobs where the g-job performance link is stronger would show larger group differences in job performance, assuming no other effects (such as selective hiring which is ubiquitous). However, hat is not the kind of data this study has. The data here are the racial proportions of each job and some information about the jobs. One cannot frame the current study as a test of SH.
However, I still think the present study is useful and I have no serious criticism of the methods used, but it's a study of something else. It's a study of what happens when one has groups with different mean ability levels and there are jobs that select for different levels of cognitive ability. In the absence of differential hiring, there should be more of the higher scoring groups in the jobs that recruit from higher up the cognitive ability distribution. Of course, we know that there is some differential hiring and using IQ tests without doing this may be illegal (but not in the military).
Preferably, one should try to model the racial proportions together based on demographic data and assumptions about the range each job recruits workers from. However, the authors prefer to take a simpler approach and check the simpler prediction that the jobs with higher means do have more persons from the higher ability groups. This is fine with me.
Write-up of regressions
The write-up of the additional regression analyses is not satisfactory. These regressions were not based on any pre-analysis hypothesizing as the writing says. They were entirely exploratory, post hoc regressions and should be clearly labeled as such. To do otherwise, is to HARK (http://psr.sagepub.com/content/2/3/196.abstract). I would write something like:
A reviewer* suggested using multiple regression by including the jobs' ratings for whether they involve working with people and things. This was done because there may be group differences in these preferences which may obscure the relationship to the mean levels of cognitive ability. A regression model was fit for each race% as the outcome and with the mean job IQ, people-interest rating and person-interest rating as the predictors. [3 tables of results] The mean IQ was a good predictor in all the models and, importantly, the small correlation seen for Asian% seemed to be due to a suppression effect from a confound with a relatively stronger preference among Asians for working with people.
I'm sorry, but the framing of the study has to be changed. This is not a test of Spearman's hypothesis.
* I usually name reviewers who give useful suggestions.
Emil, thanks for these comments.
SH: The more x requires g, the larger the race difference on x.
In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).
In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).
I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.
Obviously, the harked analyses weren’t in the original submission. We harked to “Hopefully Appease Reviewer Kirkegaard” (a double hark!?).
The analyses you suggested never occurred to us as we submitted the original paper here. So, adding the analyses had to be post hoc and exploratory. In fact we start discussion of this by stating “A reviewer recommended we explore….” I think that language makes it patently obvious that the analyses to follow are post hoc. We should add more disclaimers?
It occurs to me that “harking by reviewer” is interesting, and is likely an indictment that authors didn’t do something initially that they should have.
Also, (1) I don’t see what three regression tables add but length to the paper, and (2) I’ve always never acknowledged reviewers by name because it seems to diminish the significant service they do for our profession, for free, when they peer review.
Jensen's method (correlated vectors)
When one has multiple, preferably many, indicators of a latent trait one can use Jensen's method to check whether this latent variable is related to a criterion variable or if its the other variance components (group factors or test/item specificity). This method requires that one has multiple indicators each of which have some measurement of how well they measure the latent trait. Usually factor loadings from factor analysis are used for this purpose, but if one has item-level data, one should use item response theory-based discrimination values instead.
It is true that jobs are basically mental tests of varying difficulty. SH for this kind of data would be the claim that the jobs where the g-job performance link is stronger would show larger group differences in job performance, assuming no other effects (such as selective hiring which is ubiquitous). However, hat is not the kind of data this study has. The data here are the racial proportions of each job and some information about the jobs. One cannot frame the current study as a test of SH.
SH: The more x requires g, the larger the race difference on x.
In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).
In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).
I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.
However, I still think the present study is useful and I have no serious criticism of the methods used, but it's a study of something else. It's a study of what happens when one has groups with different mean ability levels and there are jobs that select for different levels of cognitive ability. In the absence of differential hiring, there should be more of the higher scoring groups in the jobs that recruit from higher up the cognitive ability distribution. Of course, we know that there is some differential hiring and using IQ tests without doing this may be illegal (but not in the military).
Preferably, one should try to model the racial proportions together based on demographic data and assumptions about the range each job recruits workers from. However, the authors prefer to take a simpler approach and check the simpler prediction that the jobs with higher means do have more persons from the higher ability groups. This is fine with me.
Write-up of regressions
The write-up of the additional regression analyses is not satisfactory. These regressions were not based on any pre-analysis hypothesizing as the writing says. They were entirely exploratory, post hoc regressions and should be clearly labeled as such. To do otherwise, is to HARK (http://psr.sagepub.com/content/2/3/196.abstract). I would write something like:
A reviewer* suggested using multiple regression by including the jobs' ratings for whether they involve working with people and things. This was done because there may be group differences in these preferences which may obscure the relationship to the mean levels of cognitive ability. A regression model was fit for each race% as the outcome and with the mean job IQ, people-interest rating and person-interest rating as the predictors. [3 tables of results] The mean IQ was a good predictor in all the models and, importantly, the small correlation seen for Asian% seemed to be due to a suppression effect from a confound with a relatively stronger preference among Asians for working with people.
I'm sorry, but the framing of the study has to be changed. This is not a test of Spearman's hypothesis.
* I usually name reviewers who give useful suggestions.
Obviously, the harked analyses weren’t in the original submission. We harked to “Hopefully Appease Reviewer Kirkegaard” (a double hark!?).
The analyses you suggested never occurred to us as we submitted the original paper here. So, adding the analyses had to be post hoc and exploratory. In fact we start discussion of this by stating “A reviewer recommended we explore….” I think that language makes it patently obvious that the analyses to follow are post hoc. We should add more disclaimers?
It occurs to me that “harking by reviewer” is interesting, and is likely an indictment that authors didn’t do something initially that they should have.
Also, (1) I don’t see what three regression tables add but length to the paper, and (2) I’ve always never acknowledged reviewers by name because it seems to diminish the significant service they do for our profession, for free, when they peer review.
Bryan,
In the way you frame it here, yes, that would a kind of SH test. But note that your analysis do not look at group differences directly, and neither does it look at under or over-representation directly. It only looks at representation (proportion of workings from each SIRE). I don't see how it can be a test of SH without actual group difference data being analyzed.
Of course, these extra analyses were added by my request. However, I did not advocate HARKing in the write-up.
For an example of what I had in mind, see this previous paper's review thread where another reviewer (not me!) suggested an exploratory analysis. The resulting write-up was:
(Robustness section)
During the peer review process L. J. Zigerell suggested that population density or total population might be obscuring the results. To test this, I created parallel versions of the dataset with controls. This was done simply by regressing (linear regression) each indicator on the control variable and saving the residuals. Then each control variable was used with three modes: 1) untransformed, 2) log transformed and 3) square root transformed. This was done to make the distribution more normal and suitable for the linear model used for the control. The population data was copied from Wikipedia (“List of Japanese prefectures by population,” 2015).
...
(Discussion section)
However, in an exploratory (unplanned) analysis, it was found that if one removes the effect of (the log of) population density, the usual S factor study pattern emerges: the loadings go in expected directions, S can be reliably extracted from different samples of indicators, S correlates strongly with cognitive ability, and Jensen's method indicates that the relationship is due mostly to S, not other variance.
Here's what you have:
All good.
I suggested adding the interest predictors. This writing suggests, to me at least, that there was a specific hypothesis about Asians gravitating towards people-jobs, whereas this was not the case.
I was simply saying that either people or things-interest may act as a suppressor variable, but no direction of effect was suggested. Here's the direct quote (from earlier) "Asians have substantially different job interests (no idea if this is true), which therefore throws the correlations off."
Nitpick: the true relationship not the true correlation.
Okay with me. :)
SH: The more x requires g, the larger the race difference on x.
In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).
In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).
I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.
In the way you frame it here, yes, that would a kind of SH test. But note that your analysis do not look at group differences directly, and neither does it look at under or over-representation directly. It only looks at representation (proportion of workings from each SIRE). I don't see how it can be a test of SH without actual group difference data being analyzed.
Obviously, the harked analyses weren’t in the original submission. We harked to “Hopefully Appease Reviewer Kirkegaard” (a double hark!?).
The analyses you suggested never occurred to us as we submitted the original paper here. So, adding the analyses had to be post hoc and exploratory. In fact we start discussion of this by stating “A reviewer recommended we explore….” I think that language makes it patently obvious that the analyses to follow are post hoc. We should add more disclaimers?
It occurs to me that “harking by reviewer” is interesting, and is likely an indictment that authors didn’t do something initially that they should have.
Of course, these extra analyses were added by my request. However, I did not advocate HARKing in the write-up.
For an example of what I had in mind, see this previous paper's review thread where another reviewer (not me!) suggested an exploratory analysis. The resulting write-up was:
(Robustness section)
During the peer review process L. J. Zigerell suggested that population density or total population might be obscuring the results. To test this, I created parallel versions of the dataset with controls. This was done simply by regressing (linear regression) each indicator on the control variable and saving the residuals. Then each control variable was used with three modes: 1) untransformed, 2) log transformed and 3) square root transformed. This was done to make the distribution more normal and suitable for the linear model used for the control. The population data was copied from Wikipedia (“List of Japanese prefectures by population,” 2015).
...
(Discussion section)
However, in an exploratory (unplanned) analysis, it was found that if one removes the effect of (the log of) population density, the usual S factor study pattern emerges: the loadings go in expected directions, S can be reliably extracted from different samples of indicators, S correlates strongly with cognitive ability, and Jensen's method indicates that the relationship is due mostly to S, not other variance.
Here's what you have:
A reviewer recommended we explore further the relatively weak correlation between percent Asian and IQ.
All good.
We therefore conducted multiple regression analyses testing a hypothesis that Asians gravitate more toward “people jobs” (i.e., values for People might suppress the IQ / percent Asian correlation).
I suggested adding the interest predictors. This writing suggests, to me at least, that there was a specific hypothesis about Asians gravitating towards people-jobs, whereas this was not the case.
I was simply saying that either people or things-interest may act as a suppressor variable, but no direction of effect was suggested. Here's the direct quote (from earlier) "Asians have substantially different job interests (no idea if this is true), which therefore throws the correlations off."
However, this analysis on percent Asian produced beta weights of .41 (IQ), .34 (People), and .00 (Things). It appears that values on People suppress the true correlation between Job IQ and percent Asian.
Nitpick: the true relationship not the true correlation.
Also, (1) I don’t see what three regression tables add but length to the paper, and (2) I’ve always never acknowledged reviewers by name because it seems to diminish the significant service they do for our profession, for free, when they peer review.
Okay with me. :)
Bryan,SH: The more x requires g, the larger the race difference on x.
In your example, x is job performance. The more job performance requires g, the larger the race difference. The dependent variable is job performance, measured quantitatively (e.g., units produced) or qualitatively (e.g., “excellent,” “poor”).
In my example, x is being employed in the job itself. The more being employed in a job or not depends on g, the larger the race difference. The dependent variable is “representation” (i.e., the relative percent of job holders who are White, Black, and Asian).
I do think this is a test of SH because the more IQ matters toward getting a job, the more Whites / the less Blacks there should be working that job. In other words, over/under representation in a job is partly determined by IQ, as predicted by SH, and indeed Blacks (e.g.) are more and more under-represented as job IQ goes up.
In the way you frame it here, yes, that would a kind of SH test. But note that your analysis do not look at group differences directly, and neither does it look at under or over-representation directly. It only looks at representation (proportion of workings from each SIRE). I don't see how it can be a test of SH without actual group difference data being analyzed.
You make an interesting point here, and I'm not sure I can address it.
But, suppose I subtracted the population percents for each group from the race data by job. So, for example, since 12% of the USA labor force is Black, subtracting that number from the Black column would then truly produce numbers that measure over/under representation. These numbers get more negative as job IQ goes up (Blacks are underrepresented at high-IQ jobs, presumably because of g). The reverse (i.e., a race difference) happens when looking at Whites and Asians.
Wouldn't this reversal be the "actual group difference" you're looking for above? The race difference here is in the relative percent of Blacks and Whites holding low and high IQ jobs.
If so, my idea of subtracting out the population percents seems trivial, as the correlations would remain the same values?
Would it ever be possible to test SH by looking at group differences indirectly?
I'm not sure I'm correct here, and wonder what other readers think about this study being a test of SH?
I'll reply to your other comments after we resolve this issue, as it seems key to the paper's future.
Perhaps we can get Dalliard to give his thoughts. He is also knowledgeable on the SH literature. I'll send him an email.
Would it ever be possible to test SH by looking at group differences indirectly?
I'm not sure I'm correct here, and wonder what other readers think about this study being a test of SH?
I'll reply to your other comments after we resolve this issue, as it seems key to the paper's future.
With MCV one correlates the vector of group differences with the vector of g-loadings. In this case, (1) groups differences are indexed by the employment % differences, and (2) "g-loadings" are re-conceptualized as "cognitive complexity loadings" which are indexed by the IQ -- or test score on a test that claims to measure general cognitive ability -- requirements of the jobs. Emil feels that (2) is problematic.
I disagree; Linda Gottfredson made an equivalent argument and conducted an equivalent test (and her paper passed review). We will have to wait for another reviewer to adjudicate. If they agree with Emil, you will simply have to rephrase some of the sections. What you are doing would no longer count as a valid test of SH, but would be an application of it -- a putting SH to work, as you say -- in the sense that SH would predict this -- thought so might other hypotheses -- because g is the major predictor of job differences and if there were group g differences -- as opposed to narrow ability differences or psychometric bias differences -- one would expect employment rate disparities in line with the GMA loaded-ness of Jobs.
I like the article. That there is a prediction that the % of racial composition changes as IQ job (or complexity) increases is supportive of Spearman's hypothesis, also shows (or rather, suggests) that within-correlation of IQ*job complexity may also extend to between-group context (as Gottfredson compiled lot of research showing that this correlation holds within groups, white groups) as the hypothesis expected.
I have no problem with the consistency/stability of the correlations, and their magnitude and signs indeed are supportive of the studied hypothesis.
I would appreciate if you can describe a little bit more the variables of worker activity, namely, data, people, and things, because it may not be very clear to everyone (e.g., I have only a rough idea).
You forgot a bracket.
It has little to do with the core of this study but I don't remember that Jensen said anything like this. In fact, I think Jensen's reasoning is in accordance with Dolan, in that that if there is bias, any finding in support of SH would be unreliable. They however don't disagree about the method to detect bias (but that, you and I, we already know very well).
I have no problem with the consistency/stability of the correlations, and their magnitude and signs indeed are supportive of the studied hypothesis.
I would appreciate if you can describe a little bit more the variables of worker activity, namely, data, people, and things, because it may not be very clear to everyone (e.g., I have only a rough idea).
They score six (“speaking-signaling”) on people, and two (“operating-controlling) on things.
You forgot a bracket.
I was suggesting that you might qualify your statement. For example, for precision, I might say: Spearman's hypothesis would predict that group differences are larger on more g-loaded tests, assuming no countervailing psychometric bias. Likewise: Spearman's hypothesis would predict that employment differences are larger for more g-loaded fields, assuming no countervailing societal bias e.g., affirmative action or defacto quotas. If you think that the qualification is obvious, don't bother.
It has little to do with the core of this study but I don't remember that Jensen said anything like this. In fact, I think Jensen's reasoning is in accordance with Dolan, in that that if there is bias, any finding in support of SH would be unreliable. They however don't disagree about the method to detect bias (but that, you and I, we already know very well).
© Copyright 2024 OpenPsych - All Rights Reserved