Back to [Archive] Post-review discussions

[ODP] Discounting IQ’s Relevance to Organizational Behavior: The “Somebody Else’s Pro
Given this, would you still prefer we get back into the texts and place specific quotes for some of the items?



(P3). Regarding the interpretation of the first statement (quoted in P3), I was interpreting e.g., "language fluency" differently. Presumably you are thinking of e.g., "verbal ability"; I was thinking of knowledge of the specific languages in question. Clearly, net of g, a person who could well speak both Mandarin and English would make for a better Mandarin-English language interpreter than would someone who could only speak English. This statement is ambiguous because, at least in common speech, the term "language fluency" is. Though perhaps the meaning is obvious in context to IO discussions. If so, I withdraw that claim.

Regarding the second statement, on re-reading, yes, you are correct, specifically because the author says "require" and "have to be good at".

As for the specific-general ability debate, some narrative discussions give a different slant:

Paper #1

The consensual answer to this question, however, as provided by studies and reviews on this topic is that no incremental contribution should be expected from specific cognitive abilities (Viswesvaran & Ones, 2002). Implications of this conclusion include using aggregate scores of general mental ability and disregarding specific cognitive abilities in requirement analyses and assessment
This conclusion, however, is challenged by recent metaanalytical evidence showing that specific cognitive abilities serve as additional or even more important predictors than general mental ability (Goertz et al., 2014; Lang et al., 2010; Ziegler et al., 2011). For instance, Goertz et al. revealed substantial correlations between specific cognitive abilities and training success that were comparable in strength to those usually obtained for general mental ability. Applying relative importance analysis, Lang et al. found some specific cognitive abilities to be more important for job performance than general mental ability. Finally, Ziegler et al. examined the incremental validity of specific cognitive ability in predicting training performance and found that several specific abilities were able to explain incremental variance in different training performance domains. While meta-analytic evidence thus far is based on German samples only, other primary studies with nonGerman samples further speak to a shift in paradigm (e.g., Mount, Oh, & Burns, 2008; Stanhope & Surface, 2014; Webb et al., 2007). This is not to say that all recent studies reveal evidence demonstrating that specific cognitive abilities predict work-related outcomes beyond general mental ability (e.g., Brown, Le, & Schmidt, 2006; Lievens, 2004). However, the changing picture calls for further research on moderators of the incremental contribution of specific cognitive abilities or boundary conditions of the dominant role of general mental ability. (Krumm et al. (2014). Specific cognitive abilities at work: A brief summary from two perspectives.)


Paper #2

However, there are at least two methodological issues that cast doubt on the conclusion that there is “not much more than g” (Reeve, 2004). First, much of the prior research relied on the observed subscale scores of a test battery (as defined at the discretion of test constructor) as a construct-valid surrogate for specific ability constructs (e.g., Hunter, 1986; Thorndike, 1991). By ignoring the fact that most test sub-scales confound sources of variance due to multiple specific and general abilities, these studies would have failed to obtain “clean” construct-valid assessments of specific abilities. Thus, the correlations with outcomes reflect both the variance due to ‘g’ as well as the unique variance due to the specific factor. A clear understanding of the unique value of specific abilities requires the contribution of ‘g’ to be removed (e.g., see Gustafsson, 2002, for an excellent discussion on this issue). Although Ree and Earles (1991) attempted to avoid the problem of confounded sources of variance, their reliance on atheoretical principal components is subject to the same general criticism. These principal components do not necessarily reflect any specific ability construct (as Ree & Earles, 1991 note). Moreover, recent research using analytical techniques that address the problem of the relationships between the general and specific abilities has found that the general factor accounts for much less of the explainable variance than previously believed and that the specific factors are more important in some cases (Lang, Kersting, Hülsheger, & Lang, 2010). (Reeve et al. (2015). Manifestations of intelligence: Expanding the measurement space to reconsider specific cognitive abilities.)


Apparently, the problem isn't just how intelligence is discussed in text books. Many researchers and practitioners seem to believe that specific abilities are of importance. Here were some survey results:

A survey of 703 HR scientist and practitioner members of the Society for I/O Psychology found that 82.6% of the respondents endorsed the statement, “Different jobs are likely to require different types of cognitive abilities,” and only 25.1% of the respondents endorsed the statement, “Combinations of specific aptitude tests have little advantage over measures of general cognitive ability in personnel selection,” (Murphy, Cronin, & Tam, 2003). (Schneider et al. 2015. Intelligence is multidimensional: Theoretical review and implications of specific cognitive abilities.)


It's possible that the 80% are just wrong. Alternatively, some of the differences in opinion could be due to the interpretation of the question asked:

(1) Different jobs are likely to require different types of cognitive abilities (i.e., specific abilities) for general performance (which are assessed by overall job performance measures).
(2) Different jobs are likely to require different types of cognitive abilities (i.e., specific abilities) for specific tasks (which are more assessed by job training measures).
See Schneider et al.'s (2015) model.

Anyways, because there are different readings of the literature and possibly different interpretations of the questions, I still feel that it is important to give a couple of specific examples and explanations, so readers can get a better idea of how you are grading passages. These can be placed in the supplementary file and the process should take you no more than 10 minutes.
OK, I approve publication.
Hi Chuck

I will respond to all your comments soon, but I wanted to get your opinion as to whether this appendix satisfies your request to include quotes from textbooks.

Note, this includes only IQ examples. There was only one inaccurate EQ claim in any text (which is cited in our article) and we do give examples of critical claims about EQ too (e.g., that it's perhaps the big 5 repackaged).

Thanks for your consideration.

Bryan

https://www.dropbox.com/s/a4a9emdho3za68a/appendix.docx?dl=0
Hi Chuck

I will respond to all your comments soon, but I wanted to get your opinion as to whether this appendix satisfies your request to include quotes from textbooks.

Note, this includes only IQ examples. There was only one inaccurate EQ claim in any text (which is cited in our article) and we do give examples of critical claims about EQ too (e.g., that it's perhaps the big 5 repackaged).

Thanks for your consideration.

Bryan

https://www.dropbox.com/s/a4a9emdho3za68a/appendix.docx?dl=0


I get "The link you're trying to access has expired".

Could you re-post?
Admin
Bryan,

Dropbox cannot be used as a final repository for submissions. This is because they expire and so that makes the material not available. For instance, suppose someone wants to review the review history of this paper in 5 years, s/he won't be able to find the material.

For this reason I recommend using Open Science Foundation. It has support for versioning, so it will keep track of versions.
That's odd. I know dropbox has an option to have links expire, but I didn't think I set it to do so:

https://www.dropbox.com/s/o065ebx1z4yvcut/appendix.docx?dl=0

I'll put the final version in the open source thingy.
That's odd. I know dropbox has an option to have links expire, but I didn't think I set it to do so:

https://www.dropbox.com/s/o065ebx1z4yvcut/appendix.docx?dl=0

I'll put the final version in the open source thingy.


Yes, that's fine. Thank you.
You said: "Please address P2, P3, P4, P5. I have no other comments. Thank you for your submission."

Here's our reply

(P2) The authors state:

Agreement now exists regarding the factorial structure of human mental abilities: The Cattell-Horn-Carroll (CHC) theory of intelligence (Carroll, 1993; McGrew, 2009).


I am not aware of a survey which shows this. I would rephrase the sentence along the lines of Lievens and Reeve (2012):

Most experts today accept some form of a hierarchal model, with a single general cognitive ability factor at the apex (referred to as ‘‘g’’ or ‘‘general mental ability’’) in large part due to the exhaustive work of John Carroll (1993).


This is a safer statement since there are multiple models in which g is at the apex e.g., VPR and it is seemingly not established that "most" agree with the CHC one.


Done. I will post right after this with a link to the final manuscript (?), the data file, and the appendix.


(P3) The authors state:

Consider the following three quotations, each from a different OB textbook:

--Thus, managers must decide which mental abilities are required to successfully perform each job. For example, a language interpreter…would especially need language fluency (Gibson, et al., 2012, p. 90).

--Different jobs require various blends of these [mental] abilities. As some obvious examples, writers have to be adept at word fluency, statisticians have to be good at numerical ability and numerical reasoning, and architects have to be skilled at spatial visualization (Greenberg, 2013, p. 131)."


Though misleading, I am not sure that these first two statements are in fact incorrect. Imagine that I stated, "Above and beyond g, managers must decide which specific mental abilities are required to successfully perform each job." Would you say that this was incorrect, given that specific abilities do contribute some above and beyond general ability? If not then (1) (as quoted) is not. The same can be said for (2). (I do not have the full quotes, of course.) I agree that the third statement is incorrect because of the condition "as long as". Perhaps you can find more clearly erroneous statements.


We've discussed this issue somewhat up-thread. I think we tried to use a reasonable-person / reasonable-student standard here. If I were a non-expert reading the first quote, I would reasonably assume that language fluency (and not g) is what makes one a good interpreter. This is not true.

The second quote also misleads the reasonable student into thinking that specific mental abilities are key for job success. We know this is not so.

I guess the problem is the texts often don't add the "above and beyond g" verbiage, which then makes the cite misleading at best or inaccurate (in our opinions).


(P4) You do not mention if the textbooks are at the undergraduate or graduate level. If the concern is with how IO practitioners -- who would have graduate degrees -- view IQ, this distinction is relevant. Please clarify.


OB is odd in that it's pretty much the same course for grads and undergrads (e.g., we waive it for grads if they have taken it as undergrads). The texts are interchangeable and not marketed-- as far as I know-- for grad versus undergrad. I think any grad-level course in OB (not geared toward MBAs) would feature journal articles versus a text.

(P5) Regarding the analysis, could you give a couple of examples of paragraphs/excerpts at least in the supplementary/excel file?

e.g., example excerpt -- reason coded thusly.

This gives readers/researchers a better sense of how you grade the paragraphs and what you deem to be accurate/critical, etc. This is important as perspectives may differ. For example, I would not have considered statements (1-2) in (P3) strictly to be inaccurate (as opposed to being incomplete).

[/quote]

Done-- we now refer readers in text to the appendix.
I am satisfied with the replies. Thank you. I approve publication.
Admin
You have three approvals and we can proceed with making a final edition ready for publication. Change the published date to the day you want the paper to appear. Usually, I post them, so the time zone is CET (GMT+1, Denmark).

Note that we are a collective of scholars here and we have to do all type setting and editing ourselves.
You have three approvals and we can proceed with making a final edition ready for publication. Change the published date to the day you want the paper to appear. Usually, I post them, so the time zone is CET (GMT+1, Denmark).

Note that we are a collective of scholars here and we have to do all type setting and editing ourselves.


Thanks to everyone who made comments!

Here is a dropbox folder containing the paper, appendix, and data. I'm assuming you take these files and host them at the journal? If I need to do something else, I apologize-- please let me know.

Also, I'm not sure what you mean by type setting now, so again, let me know if I have more to do here.

Thanks!
Bryan

https://www.dropbox.com/sh/skcgyr1fypei07a/AAB2QZ_pYE6IY6bfQZ6MPB-Ja?dl=0
Admin
Bryan,

You have a place in the paper where it says

_____________________

Insert Table 1 about here
_____________________


The table should be in that position, not the note.

Another thing.

Can you add a short note about author contributions, grants and conflicts of interest?
Bryan,

You have a place in the paper where it says

_____________________

Insert Table 1 about here
_____________________


The table should be in that position, not the note.

Another thing.

Can you add a short note about author contributions, grants and conflicts of interest?


I added an author note, but at the risk of sounding inept, I can't for the life of me embed the table into the text. It's landscape; the rest is portrait. It just won't paste in correctly (even when I created a blank page before the discussion section and changed that page to landscape).

Could you possibly assist?! Apologies as I've never used odt before.
Admin
Bryan,

I moved the table. To make it fit, I made the font size a bit smaller and decreased the page margins a bit.

It still needs the author contributions.
Bryan,

I moved the table. To make it fit, I made the font size a bit smaller and decreased the page margins a bit.

It still needs the author contributions.


Thanks much for your help.

Here is a link to the folder with the final manuscript, appendix, and data file.

https://www.dropbox.com/sh/skcgyr1fypei07a/AAB2QZ_pYE6IY6bfQZ6MPB-Ja?dl=0
Admin
Bryan,

I edited the paper slightly for various things:
- Changed all section headers to font 14.
- Removed space after section headers.
- Moved the author notes to the back before references, changed name to "Author contributions and funding". Changed text slightly (essay → article).
- Publication type is research article.
- Decreased font size of main title a bit so that it fits on 2 lines.
- Changed "misperceptions" in the abstract to "misconceptions", like was previously done for the introduction.
- Some other small changes.

Can you let me know if the PDF is how you want it?
Bryan,

I edited the paper slightly for various things:
- Changed all section headers to font 14.
- Removed space after section headers.
- Moved the author notes to the back before references, changed name to "Author contributions and funding". Changed text slightly (essay → article).
- Publication type is research article.
- Decreased font size of main title a bit so that it fits on 2 lines.
- Changed "misperceptions" in the abstract to "misconceptions", like was previously done for the introduction.
- Some other small changes.

Can you let me know if the PDF is how you want it?


Thanks for the time you spent on it. I approve!
Admin
I had a native English speaker read it through. She did not find any errors.
Admin
I added a small section with a note about the peer review thread (this one) and the supplementary material repository. https://osf.io/r8q7m/ This will stay even if this website goes down and all backups are lost (hope not!).

The paper has been published here: http://openpsych.net/ODP/2015/05/discounting-iqs-relevance-to-organizational-behavior-the-somebody-elses-problem-in-management-education/

Let me know if the authors are satisfied with this. Then I can move this thread to the post-publication forum.

One last thing. The earlier versions of the submission are now not available, but they should be. Can you either upload them here or email them to me? Then I will add them to the OSF site.

I have promoted the paper on Twitter here: https://twitter.com/OpenPsychJour/status/603039669316620288 I recommend getting on Twitter as a way to promote your own research and talk with other researchers. I did not see a Twitter account for the first author. Let me know if any author has one, so we can tag you. :)
I added a small section with a note about the peer review thread (this one) and the supplementary material repository. https://osf.io/r8q7m/ This will stay even if this website goes down and all backups are lost (hope not!).

The paper has been published here: http://openpsych.net/ODP/2015/05/discounting-iqs-relevance-to-organizational-behavior-the-somebody-elses-problem-in-management-education/

Let me know if the authors are satisfied with this. Then I can move this thread to the post-publication forum.

One last thing. The earlier versions of the submission are now not available, but they should be. Can you either upload them here or email them to me? Then I will add them to the OSF site.

I have promoted the paper on Twitter here: https://twitter.com/OpenPsychJour/status/603039669316620288 I recommend getting on Twitter as a way to promote your own research and talk with other researchers. I did not see a Twitter account for the first author. Let me know if any author has one, so we can tag you. :)


I put the first three versions in the drop box folder. Thanks again. I am not a big twitter fan-- I used to have an account and then realized I have nothing profound to say;)

https://www.dropbox.com/sh/skcgyr1fypei07a/AAB2QZ_pYE6IY6bfQZ6MPB-Ja?dl=0


eta: the tweet has my name spelled wrong...