Journal:
Open Quantitative Sociology & Political Science
Authors:
Emil O. W. Kirkegaard
Julius D. Bjerrekær
Noah Carl
Title:
Cognitive ability and political preferences in Denmark
Abstract:
We investigated the relationships between cognitive ability and multi-dimensional political preferences in a recent Danish sample (n = 320). Respondents answered 10 questions pertaining to specific social issues, 10 questions pertaining to specific economic issues, as well as taking 4 cognitive items. They had previously taken a 5-item cognitive test (ICAR5), and been asked to rate themselves on social liberalism and economic liberalism. We documented a general factor of social liberalism across the questions on social issues, and a general factor of economic liberalism across the questions on economic issues. Self-assessed social liberalism had a small positive correlation with measured social liberalism (r = .12), while self-assessed economic liberalism had a moderate positive correlation with measured economic liberalism (r = .47). These findings were in line with our predictions. Contrary to our predictions, however, social liberalism and economic liberalism had a weak positive correlation (r = .10; 95% CI = [–.04, .22]) and cognitive ability was practically unrelated to both social liberalism (r = .06; 95% CI = [–.07, .19]) and economic liberalism (r = .05; 95% CI = [–.08, .18]).
Key words:
cognitive ability, intelligence, IQ, political preferences, political dimensions, Denmark, open data, preregistered, social liberalism, social conservatism, economic liberalism
Length:
~3800 words, 21 pages.
Files:
https://osf.io/xdpcq/
Note that the questionnaire is currently not available. It will be made available shortly (pending email reply from the survey handler).
External reviewers:
We will attempt to recruit an external reviewer who's familiar with this area. Perhaps Peter Kurrild-Klitgaard. Prof. Pol. Sci. @ Copenhagen Uni. http://forskning.ku.dk/find-en-forsker/?pure=da/persons/63119
Since this paper has two of the internal reviewers familiar with this topic, we probably need to recruit an additional temp. external reviewer or a new internal one.
Status
The results in the paper are not final because we are still collecting a little bit of data (perhaps 30 more cases). However, it is very unlikely that they will substantially change. For this reason, the numbers in the paper are temporary and will be updated.
The tables are temporary. Julius will typeset the paper in LaTeX, so there is no reason with bothering to make the tables in Word pretty. But they should be readable for the sake of review.
Back to [Archive] Post-review discussions
Grammar-type comments are in the attachment. More substantively:
1. It might improve the paper if the text explicitly indicated what research design elements were preregistered and what elements were not. Some of the text implies exploratory analyses (e.g., "we tried conditioning on..."), but ideally readers would not need to consult the preregistration plan to know what parts of the research design were preregistered.
2. The "Exclusions" section mentioned that about 20% of the sample failed at least one attention-check-type item, but the section did not mention whether these respondents were included in the analysis. It would be a good idea to explicitly indicate in this section whether respondents who failed a check were removed from the sample. I also think that the attention-check-type items might have been absent from the preregistration plan; if that's the case, then it might be worth noting this in the text.
3. The conclusion indicates that the results that differ from prior research might be attributable to the sample not being representative, but there appears to be no discussion about how representative the original sample or the follow-up study's sample might be of the population of interest. The paper might benefit from a discussion and/or analysis of how representative the follow-up sample is and a discussion of the extent to which the present study should be preferred or discounted relative to prior research that had a different result. It would also be worth quantifying in the conclusion how much the present study's results differ from results of prior research, such as by noting the cognitive ability/social liberalism correlations from Carl 2015b and Onraet et al. 2015. The 95% CI for the present research appears to be somewhat close to the 0.10 to 0.30 range quoted in the introduction.
1. It might improve the paper if the text explicitly indicated what research design elements were preregistered and what elements were not. Some of the text implies exploratory analyses (e.g., "we tried conditioning on..."), but ideally readers would not need to consult the preregistration plan to know what parts of the research design were preregistered.
2. The "Exclusions" section mentioned that about 20% of the sample failed at least one attention-check-type item, but the section did not mention whether these respondents were included in the analysis. It would be a good idea to explicitly indicate in this section whether respondents who failed a check were removed from the sample. I also think that the attention-check-type items might have been absent from the preregistration plan; if that's the case, then it might be worth noting this in the text.
3. The conclusion indicates that the results that differ from prior research might be attributable to the sample not being representative, but there appears to be no discussion about how representative the original sample or the follow-up study's sample might be of the population of interest. The paper might benefit from a discussion and/or analysis of how representative the follow-up sample is and a discussion of the extent to which the present study should be preferred or discounted relative to prior research that had a different result. It would also be worth quantifying in the conclusion how much the present study's results differ from results of prior research, such as by noting the cognitive ability/social liberalism correlations from Carl 2015b and Onraet et al. 2015. The 95% CI for the present research appears to be somewhat close to the 0.10 to 0.30 range quoted in the introduction.
Good criticism. We are working on revision.
Many thanks for the review.
Grammar errors have been changed.
Analyses that were not pre-registered have been indicated as such in the text.
It is now clearly stated in the text that all respondents who failed at least one of the control questions were excluded from the analyses.
A comparison of the Danish adult population, the original sample and our sub-sample on age, sex and education has been included in the Discussion section. Onraet et al.'s (2015) meta-analytic effect size has been reported in the text. And it has been noted in the text that the upper bound of the confidence interval for the correlation between cognitive ability and social liberalism is equal to this effect size.
Grammar-type comments are in the attachment.
Grammar errors have been changed.
1. It might improve the paper if the text explicitly indicated what research design elements were preregistered and what elements were not. Some of the text implies exploratory analyses (e.g., "we tried conditioning on..."), but ideally readers would not need to consult the preregistration plan to know what parts of the research design were preregistered.
Analyses that were not pre-registered have been indicated as such in the text.
2. The "Exclusions" section mentioned that about 20% of the sample failed at least one attention-check-type item, but the section did not mention whether these respondents were included in the analysis. It would be a good idea to explicitly indicate in this section whether respondents who failed a check were removed from the sample. I also think that the attention-check-type items might have been absent from the preregistration plan; if that's the case, then it might be worth noting this in the text.
It is now clearly stated in the text that all respondents who failed at least one of the control questions were excluded from the analyses.
3. The conclusion indicates that the results that differ from prior research might be attributable to the sample not being representative, but there appears to be no discussion about how representative the original sample or the follow-up study's sample might be of the population of interest. The paper might benefit from a discussion and/or analysis of how representative the follow-up sample is and a discussion of the extent to which the present study should be preferred or discounted relative to prior research that had a different result. It would also be worth quantifying in the conclusion how much the present study's results differ from results of prior research, such as by noting the cognitive ability/social liberalism correlations from Carl 2015b and Onraet et al. 2015. The 95% CI for the present research appears to be somewhat close to the 0.10 to 0.30 range quoted in the introduction.
A comparison of the Danish adult population, the original sample and our sub-sample on age, sex and education has been included in the Discussion section. Onraet et al.'s (2015) meta-analytic effect size has been reported in the text. And it has been noted in the text that the upper bound of the confidence interval for the correlation between cognitive ability and social liberalism is equal to this effect size.
Two points:
1. All too often, the concepts of "social liberalism" and "economic liberalism" are disingenuous. Many people are liberal because they see liberalism as a means to weaken the existing system or the existing elite. Once these "liberals" gain power, they develop an interest in the state as a means to enforce their values, or as a supposed "temporary measure" to keep the other side from regaining power. It might be useful to question the subjects to find out whether they see liberalism as an end or as a means to an end. Do the subjects themselves accept your dichotomy of liberalism vs. non-liberalism?
2. Were there gender differences in the results? Usually, when the right or the left is on the upswing, women tend to lag behind men.
1. All too often, the concepts of "social liberalism" and "economic liberalism" are disingenuous. Many people are liberal because they see liberalism as a means to weaken the existing system or the existing elite. Once these "liberals" gain power, they develop an interest in the state as a means to enforce their values, or as a supposed "temporary measure" to keep the other side from regaining power. It might be useful to question the subjects to find out whether they see liberalism as an end or as a means to an end. Do the subjects themselves accept your dichotomy of liberalism vs. non-liberalism?
2. Were there gender differences in the results? Usually, when the right or the left is on the upswing, women tend to lag behind men.
Peter,
1.
We can't ask the subjects more questions because the data collection is complete. As you can see in the paper, we are pretty clear about what our concepts are because we included all the 10 questions used to measure each in the paper. No verbal tricks. The subjects were not told about the purpose of the political questions, so the concept of liberalness never came up. In the previous survey where we asked about their self-rated stances, we did not use the word "liberal" either. We asked them about how much they prefer personal freedom and economic freedom with some examples of each (e.g. legalized cannabis, lower taxes).
In general, I prefer to stay away from contested political labels.
2.
As you can see in the regression tables, men were somewhat more freedom-preferring for both axes (controlling for other things). Standardized betas of .30 and .47. These, however, did not survive the LASSO regression. Possibly because we had to treat them as continuous variables for the purpose of that modeling. This deflates the results a bit, but I think we can be pretty certain that the sex difference is real. I did try to find a LASSO method that worked with GLM, so one can use a categorical variable, but did not get it to work properly.
I see that we forgot to include the simple bivariate results, i.e. correlations/d values for all the predictors with both political axes. We will revise to include that since it's important information.
http://rpubs.com/EmilOWK/208757
1.
We can't ask the subjects more questions because the data collection is complete. As you can see in the paper, we are pretty clear about what our concepts are because we included all the 10 questions used to measure each in the paper. No verbal tricks. The subjects were not told about the purpose of the political questions, so the concept of liberalness never came up. In the previous survey where we asked about their self-rated stances, we did not use the word "liberal" either. We asked them about how much they prefer personal freedom and economic freedom with some examples of each (e.g. legalized cannabis, lower taxes).
In general, I prefer to stay away from contested political labels.
2.
As you can see in the regression tables, men were somewhat more freedom-preferring for both axes (controlling for other things). Standardized betas of .30 and .47. These, however, did not survive the LASSO regression. Possibly because we had to treat them as continuous variables for the purpose of that modeling. This deflates the results a bit, but I think we can be pretty certain that the sex difference is real. I did try to find a LASSO method that worked with GLM, so one can use a categorical variable, but did not get it to work properly.
I see that we forgot to include the simple bivariate results, i.e. correlations/d values for all the predictors with both political axes. We will revise to include that since it's important information.
http://rpubs.com/EmilOWK/208757
The pollster forgot to invite the 66-80 year olds for this study. He will do so now and this will probably increase the sample size by a bit, maybe 10-30 persons. It should also help with the age representation problem: currently mean age of 37.2 in this sample vs. 49.1 in 18-80 population Denmark.
It is unlikely to change the main findings because age does not seem to interact with CA for predicting political preferences.
It is unlikely to change the main findings because age does not seem to interact with CA for predicting political preferences.
I checked the revision and approve the submission.
For what it's worth, I think that "measure" should be "measured" in the passage "the correlation between self-assessed and measure social liberalism" (p. 14). Also, it's not a problem, but the figures use "personal freedom" and "economic freedom" for phenomena that the text refers to with "social liberalism" and "economic liberalism".
For what it's worth, I think that "measure" should be "measured" in the passage "the correlation between self-assessed and measure social liberalism" (p. 14). Also, it's not a problem, but the figures use "personal freedom" and "economic freedom" for phenomena that the text refers to with "social liberalism" and "economic liberalism".
Good comment.
The reason for the slight inconsistencies in nomenclature is that I wrote the R code and thus the figure captions, and Noah wrote the paper. I will make it more consistent and fix the error you found.
The reason for the slight inconsistencies in nomenclature is that I wrote the R code and thus the figure captions, and Noah wrote the paper. I will make it more consistent and fix the error you found.
[font=Calibri, serif]In the abstract there is a general description of what was done, but no clear statement of objective. I think most readers would want to see something that clearly states something about the purpose of the study.
[align=justify][font=Calibri, serif]I am confused by this:[/align]
[align=justify][font=Times New Roman, serif]By going through a pollster, we administered 24 additional items to the sample: [/align]
[font=Calibri, serif]After reading again, I assume that this simply means that when you used the pollster, you added the additional items. If so, the confusion I experienced was related to the word "by." It could be written to say "When the polling was repeated with a pollster, we added 24 additional items." ... Or something to that effect.
[align=justify][font=Times New Roman, serif]The final sample size was [font=Times New Roman, serif]n[font=Times New Roman, serif][u] [/u][font=Times New Roman, serif]327 (65% of the original [font=Times New Roman, serif]n[font=Times New Roman, serif][u])[/u][font=Times New Roman, serif][u].[/u][/align]
[font=Calibri, serif]On first reading, it was unclear as to why the final sample size was smaller. I now assume that 327 was the number of people who responded to the pollsters, although it was not obvious to me that they contacted all of the first group or only part. Can this be worded differently?
[font=Calibri, serif]Later (section 2.3) there is a comment about removing responses where random answering was indicated, but that resulted in a 20% reduction. Was the 20% reduction before or after the initial reduction? I assume that is what happened.
[align=justify][font=Times New Roman, serif]As Table indicates, responders were slightly younger, had slightly lower cognitive ability and were slightly less educated than non-responders. There was, therefore, some selection bias in responding to our survey.[/align]
[font=Calibri, serif]Some readers will recall that the usual pattern of participation is skewed towards brighter people, due to their greater willingness to participate. This case went the other way. I assume this is random, but it might be worth a sentence or two to explain the likely cause of this somewhat unexpected response makeup.
[font=Calibri, serif]Just a comment on the Likert scales: It seems to me that by dividing the scales into two ranges, an unnecessary distraction resulted. This meant lots of discussion about combining the results when there was nothing gained with respect to the paper. Using either one would have, in my opinion, been better and would not have resulted in any loss of value. Given that this was already done, there is no way to change it now.
[font=Calibri, serif]Figures 3 and 4
[font=Calibri, serif]Some of the item characteristic curves seem to have low slopes, making the inflection point indistinct. I understand that you can define the latent trait despite this, but it appears to me that those curves are not what you would want. The factor analysis shows 4 distributions that show reasonable looking shapes, but the others seem to be uninformative. There was little discussion of this, but there was a mention of too few easy items. Can more be said about the impact of this skew on the outcome? It leaves me wondering how one can justify using uneven item difficulty to indicate the relations that are discusses with respect to population preferences versus intelligence.
[align=justify][font=Times New Roman, serif]Surprisingly, there wasn’t much of a relationship in either case, not even a non-linear one: although both estimates are positive, their confidence intervals encompass [font=Times New Roman, serif]zero.[/align]
[align=justify][font=Times New Roman, serif]and cognitive ability was practically unrelated to both social liberalism ([font=Times New Roman, serif]r[font=Times New Roman, serif] = .08; 95% CI = [–.04, .20])[font=Times New Roman, serif] and economic liberalism ([font=Times New Roman, serif]r[font=Times New Roman, serif] = .06; 95% CI = [–.06, .18][font=Times New Roman, serif]); although both of these correlations were in the expected direction, they were much smaller than we had anticipated.[/align]
[align=justify][font=Times New Roman, serif]Second, our measure of cognitive ability was somewhat crude, comprising just 9 items in total, and providing limited discriminability in the left-hand tail. [/align]
[font=Calibri, serif]The above may be related to my comments concerning figures 3 and 4. The papers I have read on the subject of political orientation on left-right scale or liberal-conservative scales have shown that there is a correlation with intelligence (as you acknowledge near the end of the paper), although the results have been somewhat mixed. So, I too am surprised by this result and ask if it can be traced to the failure to use a population representative measurement of intelligence. I am a bit more confused in that the ICAR5 should have given an adequate measurement. I admit to confusion, which may be entirely mine and not a problem with the methodology.
[align=justify][font=Calibri, serif]I am confused by this:[/align]
[align=justify][font=Times New Roman, serif]By going through a pollster, we administered 24 additional items to the sample: [/align]
[font=Calibri, serif]After reading again, I assume that this simply means that when you used the pollster, you added the additional items. If so, the confusion I experienced was related to the word "by." It could be written to say "When the polling was repeated with a pollster, we added 24 additional items." ... Or something to that effect.
[align=justify][font=Times New Roman, serif]The final sample size was [font=Times New Roman, serif]n[font=Times New Roman, serif][u] [/u][font=Times New Roman, serif]327 (65% of the original [font=Times New Roman, serif]n[font=Times New Roman, serif][u])[/u][font=Times New Roman, serif][u].[/u][/align]
[font=Calibri, serif]On first reading, it was unclear as to why the final sample size was smaller. I now assume that 327 was the number of people who responded to the pollsters, although it was not obvious to me that they contacted all of the first group or only part. Can this be worded differently?
[font=Calibri, serif]Later (section 2.3) there is a comment about removing responses where random answering was indicated, but that resulted in a 20% reduction. Was the 20% reduction before or after the initial reduction? I assume that is what happened.
[align=justify][font=Times New Roman, serif]As Table indicates, responders were slightly younger, had slightly lower cognitive ability and were slightly less educated than non-responders. There was, therefore, some selection bias in responding to our survey.[/align]
[font=Calibri, serif]Some readers will recall that the usual pattern of participation is skewed towards brighter people, due to their greater willingness to participate. This case went the other way. I assume this is random, but it might be worth a sentence or two to explain the likely cause of this somewhat unexpected response makeup.
[font=Calibri, serif]Just a comment on the Likert scales: It seems to me that by dividing the scales into two ranges, an unnecessary distraction resulted. This meant lots of discussion about combining the results when there was nothing gained with respect to the paper. Using either one would have, in my opinion, been better and would not have resulted in any loss of value. Given that this was already done, there is no way to change it now.
[font=Calibri, serif]Figures 3 and 4
[font=Calibri, serif]Some of the item characteristic curves seem to have low slopes, making the inflection point indistinct. I understand that you can define the latent trait despite this, but it appears to me that those curves are not what you would want. The factor analysis shows 4 distributions that show reasonable looking shapes, but the others seem to be uninformative. There was little discussion of this, but there was a mention of too few easy items. Can more be said about the impact of this skew on the outcome? It leaves me wondering how one can justify using uneven item difficulty to indicate the relations that are discusses with respect to population preferences versus intelligence.
[align=justify][font=Times New Roman, serif]Surprisingly, there wasn’t much of a relationship in either case, not even a non-linear one: although both estimates are positive, their confidence intervals encompass [font=Times New Roman, serif]zero.[/align]
[align=justify][font=Times New Roman, serif]and cognitive ability was practically unrelated to both social liberalism ([font=Times New Roman, serif]r[font=Times New Roman, serif] = .08; 95% CI = [–.04, .20])[font=Times New Roman, serif] and economic liberalism ([font=Times New Roman, serif]r[font=Times New Roman, serif] = .06; 95% CI = [–.06, .18][font=Times New Roman, serif]); although both of these correlations were in the expected direction, they were much smaller than we had anticipated.[/align]
[align=justify][font=Times New Roman, serif]Second, our measure of cognitive ability was somewhat crude, comprising just 9 items in total, and providing limited discriminability in the left-hand tail. [/align]
[font=Calibri, serif]The above may be related to my comments concerning figures 3 and 4. The papers I have read on the subject of political orientation on left-right scale or liberal-conservative scales have shown that there is a correlation with intelligence (as you acknowledge near the end of the paper), although the results have been somewhat mixed. So, I too am surprised by this result and ask if it can be traced to the failure to use a population representative measurement of intelligence. I am a bit more confused in that the ICAR5 should have given an adequate measurement. I admit to confusion, which may be entirely mine and not a problem with the methodology.
Bob,
Thanks for looking over the study.
I have rewritten the abstract to briefly note why we carried out the study. I added: []Multiple studies have reported positive relationships between cognitive ability and preferences for freedom, both at the personal level (e.g. drug use) and the economic (e.g. smaller government). To add to this, we investigated the ...
I have rewritten the text in multiple places to be more clear.
What was done was that:
An earlier study surveyed some n=552 Danes on various variables for the stereotype study (). We arranged with the pollster to send out invitations to take part in a second survey to the same 550 persons. Some of them had either quit taking part in surveys all together or simply did not want or have time to take part in our second survey. For this reason, the sample size is smaller than the one before at n=333. Some people failed one or (usually) more of the control questions and due to an error at the pollster, some could not be linked up between the two surveys. This left us with n=259 persons with complete data for the analyses.
[font=Calibri,serif]
I considered it, but since the difference is small (d=23), the reversed finding probably reflects a chance happening and is not worth commenting on.
[font=Calibri,serif]
I agree, but we didn't know that before carrying out the study! :) Now we know that it doesn't matter whether one uses 7-point or 101-point scales.
[font=Calibri,serif]
The figures (one was missing and has been added) are primarily for the readers more expert in IRT. However, I have rewritten some of the text to be more informative. The missing figure was the one showing the sum of the information from the items, which makes it clear that there was a lack of discriminative ability on the left tail. But in general, it was not that large a problem: mean pass rate was 37% with the very difficult item and 41% without which is pretty close to the optimal value of 50%.
I realize that the IRT figures are ugly. However, it is not easy to produce better ones because the figures are generated by the package that calculates the IRT results (psych) and they are based on the archaic base plotting system, not the modern ggplot2 system.
[font=Calibri,serif]
The somewhat mixed results comes from some researchers trying to model political preferences with only 1 dimension (liberal-conservative). The mixed results to some degree reflect the different ways different researchers have operationalized this supposed 1-dimensional trait. If they used many items related to 'social conservatism' (e.g. opposition to legalized abortion) they obtain negative correlations, but if they use fiscal/economic measures, the relationships turn positive. If they used some kind of mix, the results will be unclear.
--
As it happens, there had been a bug in the IRT software (scoreIrt(), https://cran.r-project.org/web/packages/psych/news.html). The bug meant that the scores from IRT were incorrect and was especially problematic for short scales (such as ours), which affected the cognitive ability estimates from this study. The bug has now been corrected and this increased the correlations to the point that they are more in line with the theoretical expectations, especially after correction for measurement error. In fact, most of the tables and figures had to be replaced because of this error.
This also meant that much of the discussion had to be rewritten which I have done. I have also inserted proper tables in the paper as well and prettier and better labelled figures.
I have updated the files.
Thanks for looking over the study.
[font=Calibri,serif]In the abstract there is a general description of what was done, but no clear statement of objective. I think most readers would want to see something that clearly states something about the purpose of the study.
I have rewritten the abstract to briefly note why we carried out the study. I added: []Multiple studies have reported positive relationships between cognitive ability and preferences for freedom, both at the personal level (e.g. drug use) and the economic (e.g. smaller government). To add to this, we investigated the ...
[font=Calibri,serif]I am confused by this:
[align=justify][font=Times New Roman,serif]By going through a pollster, we administered 24 additional items to the sample: [/align]
[font=Calibri,serif]After reading again, I assume that this simply means that when you used the pollster, you added the additional items. If so, the confusion I experienced was related to the word "by." It could be written to say "When the polling was repeated with a pollster, we added 24 additional items." ... Or something to that effect.
[align=justify][font=Times New Roman,serif]The final sample size was [font=Times New Roman,serif]n[font=Times New Roman,serif][u] [/u][font=Times New Roman,serif]327 (65% of the original [font=Times New Roman,serif]n[font=Times New Roman,serif][u])[/u][font=Times New Roman,serif][u].[/u][/align]
[font=Calibri,serif]On first reading, it was unclear as to why the final sample size was smaller. I now assume that 327 was the number of people who responded to the pollsters, although it was not obvious to me that they contacted all of the first group or only part. Can this be worded differently?
[font=Calibri,serif]Later (section 2.3) there is a comment about removing responses where random answering was indicated, but that resulted in a 20% reduction. Was the 20% reduction before or after the initial reduction? I assume that is what happened.
I have rewritten the text in multiple places to be more clear.
What was done was that:
An earlier study surveyed some n=552 Danes on various variables for the stereotype study (). We arranged with the pollster to send out invitations to take part in a second survey to the same 550 persons. Some of them had either quit taking part in surveys all together or simply did not want or have time to take part in our second survey. For this reason, the sample size is smaller than the one before at n=333. Some people failed one or (usually) more of the control questions and due to an error at the pollster, some could not be linked up between the two surveys. This left us with n=259 persons with complete data for the analyses.
[font=Times New Roman,serif]As Table indicates, responders were slightly younger, had slightly lower cognitive ability and were slightly less educated than non-responders. There was, therefore, some selection bias in responding to our survey.
[font=Calibri,serif]Some readers will recall that the usual pattern of participation is skewed towards brighter people, due to their greater willingness to participate. This case went the other way. I assume this is random, but it might be worth a sentence or two to explain the likely cause of this somewhat unexpected response makeup.
[font=Calibri,serif]
I considered it, but since the difference is small (d=23), the reversed finding probably reflects a chance happening and is not worth commenting on.
[font=Calibri,serif]Just a comment on the Likert scales: It seems to me that by dividing the scales into two ranges, an unnecessary distraction resulted. This meant lots of discussion about combining the results when there was nothing gained with respect to the paper. Using either one would have, in my opinion, been better and would not have resulted in any loss of value. Given that this was already done, there is no way to change it now.
[font=Calibri,serif]
I agree, but we didn't know that before carrying out the study! :) Now we know that it doesn't matter whether one uses 7-point or 101-point scales.
[font=Calibri,serif]Figures 3 and 4
[font=Calibri,serif]Some of the item characteristic curves seem to have low slopes, making the inflection point indistinct. I understand that you can define the latent trait despite this, but it appears to me that those curves are not what you would want. The factor analysis shows 4 distributions that show reasonable looking shapes, but the others seem to be uninformative. There was little discussion of this, but there was a mention of too few easy items. Can more be said about the impact of this skew on the outcome? It leaves me wondering how one can justify using uneven item difficulty to indicate the relations that are discusses with respect to population preferences versus intelligence.
[font=Calibri,serif]
The figures (one was missing and has been added) are primarily for the readers more expert in IRT. However, I have rewritten some of the text to be more informative. The missing figure was the one showing the sum of the information from the items, which makes it clear that there was a lack of discriminative ability on the left tail. But in general, it was not that large a problem: mean pass rate was 37% with the very difficult item and 41% without which is pretty close to the optimal value of 50%.
I realize that the IRT figures are ugly. However, it is not easy to produce better ones because the figures are generated by the package that calculates the IRT results (psych) and they are based on the archaic base plotting system, not the modern ggplot2 system.
[font=Calibri,serif]The above may be related to my comments concerning figures 3 and 4. The papers I have read on the subject of political orientation on left-right scale or liberal-conservative scales have shown that there is a correlation with intelligence (as you acknowledge near the end of the paper), although the results have been somewhat mixed. So, I too am surprised by this result and ask if it can be traced to the failure to use a population representative measurement of intelligence. I am a bit more confused in that the ICAR5 should have given an adequate measurement. I admit to confusion, which may be entirely mine and not a problem with the methodology.
[font=Calibri,serif]
The somewhat mixed results comes from some researchers trying to model political preferences with only 1 dimension (liberal-conservative). The mixed results to some degree reflect the different ways different researchers have operationalized this supposed 1-dimensional trait. If they used many items related to 'social conservatism' (e.g. opposition to legalized abortion) they obtain negative correlations, but if they use fiscal/economic measures, the relationships turn positive. If they used some kind of mix, the results will be unclear.
--
As it happens, there had been a bug in the IRT software (scoreIrt(), https://cran.r-project.org/web/packages/psych/news.html). The bug meant that the scores from IRT were incorrect and was especially problematic for short scales (such as ours), which affected the cognitive ability estimates from this study. The bug has now been corrected and this increased the correlations to the point that they are more in line with the theoretical expectations, especially after correction for measurement error. In fact, most of the tables and figures had to be replaced because of this error.
This also meant that much of the discussion had to be rewritten which I have done. I have also inserted proper tables in the paper as well and prettier and better labelled figures.
I have updated the files.
Emil - The changes you made seem to correct the points that were unclear. Some of the changes were a great improvement. I will download and read the revised text. I assume it will be good and, if so, I will approve.
I approve the paper for publication. I would like to offer one additional suggestion:
which meant that it provided less discriminability in the left-hand tail.
The word "discriminability" is awkward. Perhaps you could find alternate words. For example... "it provided lower resolution in the left-hand tail." Or "it was less sensitive to detecting differences in..." Or "it did not provide the desired level of discrimination in.."
which meant that it provided less discriminability in the left-hand tail.
The word "discriminability" is awkward. Perhaps you could find alternate words. For example... "it provided lower resolution in the left-hand tail." Or "it was less sensitive to detecting differences in..." Or "it did not provide the desired level of discrimination in.."
Given the current social attention to various forms of undesired or allegedly undesired forms of discrimination, the word is somewhat odd to use. However, this is the standard term used in IRT and we merely follow normal practice. See e.g. https://www.goodreads.com/book/show/11071705-item-response-theory
I quote from the introduction:
Now, we used the variant discriminability to avoid the precise version of the word. This is also done by many others as a quick search will show.
I quote from the introduction:
Item Discrimination
A higher discrimination means that the item differentiates (discriminates) between examinees with different levels of the construct. Thus, high discrimination is desirable. The purpose of using the instrument is to differentiate (discriminate) between examinees who know the material tested and those who do not, or on an attitude scale, between those who have positive attitudes and those who have negative attitudes. In CTT, the corrected item-total point-biserial correlation is the typical index of discrimination; when this is positive, examinees who answer the item correctly (or endorse the item) score higher on the sum of the remaining items than do those who answer the item incorrectly (or disagree with the item). In IRT, an index symbolized as a is a measure of the item discrimination. This index is sometimes called the slope, because it indicates how steeply the probability of correct response changes as the proficiency or trait increases. In both CTT and IRT, higher values indicate greater discrimination.
Now, we used the variant discriminability to avoid the precise version of the word. This is also done by many others as a quick search will show.
Now, we used the variant discriminability to avoid the precise version of the word. This is also done by many others as a quick search will show.
Okay, that is fine with me. It was only a comment.
This paper could be better organized. The Introduction should be longer with an explanation of the enigmatic sentence "Many studies have examined the relationship between cognitive ability and political preferences." At the very least, you should provide some references for that sentence. Who else has studied this relationship? What did they find? At one point, you mention Steven Pinker. Has he written on this specific subject?
The working hypotheses should be stated in the Introduction, and not further in the text. These are in fact stated on p. 22:
- Cognitive ability will be positively related to the freedom end of both axes.
- There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis.
These are interesting hypotheses. What is the rationale for them? Are they based on previous findings in the literature? Are they a personal hunch? Please tell us.
[]The Results section should be shorter. Much of the data analysis could be summarized or placed in an appendix. The conclusions should likewise be summarized and not provided piecemeal. The main conclusion seems to appear on p. 45: "In general, the correlations are near zero as expected, but there are a few exceptions. Smarter people seem to be somewhat against the government unsuring jobs [reducing job security?] higher minimum wages, and somewhat for blasphemy and flag burning being legal[][font=sans-serif]."
There should be a Discussion section. Why are the correlations so low? How does this finding compare with other findings in the literature?
The working hypotheses should be stated in the Introduction, and not further in the text. These are in fact stated on p. 22:
- Cognitive ability will be positively related to the freedom end of both axes.
- There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis.
These are interesting hypotheses. What is the rationale for them? Are they based on previous findings in the literature? Are they a personal hunch? Please tell us.
[]The Results section should be shorter. Much of the data analysis could be summarized or placed in an appendix. The conclusions should likewise be summarized and not provided piecemeal. The main conclusion seems to appear on p. 45: "In general, the correlations are near zero as expected, but there are a few exceptions. Smarter people seem to be somewhat against the government unsuring jobs [reducing job security?] higher minimum wages, and somewhat for blasphemy and flag burning being legal[][font=sans-serif]."
There should be a Discussion section. Why are the correlations so low? How does this finding compare with other findings in the literature?
Thanks Peter.
We'll work on a revision.
We'll work on a revision.
Peter,
Did you perhaps read an old version of the submission? The one I'm looking at does not seem to match the one you comment about. E.g. there is no "many studies have" phrase.
Maybe I made a mistake. I have re-uploaded the files. It's the paper.docx/paper.pdf files.
https://osf.io/xdpcq/files/
Did you perhaps read an old version of the submission? The one I'm looking at does not seem to match the one you comment about. E.g. there is no "many studies have" phrase.
Maybe I made a mistake. I have re-uploaded the files. It's the paper.docx/paper.pdf files.
https://osf.io/xdpcq/files/
Yes, I read doc.docx instead of paper.docx.
This version is much better organized and I approve its publication.
A few minor criticisms :
1. The first hypothesis is clearly stated in the Introduction but the second hypothesis appears in a rather vague form:
original version: "There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis."
current version: "[][font=sans-serif]Cognitive ability appears to have a non-monotonic relation to measures of [][font=sans-serif]economically liberal beliefs ..."
[]Why did you change the second hypothesis?
[]2.[] "rate themselves on social liberalism and economic liberalism" can be changed to "rate their degree of adherence to social and economic liberalism"
[]3. I'm not sure why the correlation is weaker with social liberalism than with economic liberalism. This point should be addressed in the Discussion section. I can think of several possible reasons. Social liberalism has a weaker theoretical basis and may be perceived as less "scientific" than economic liberalism. Social issues tend to be more controversial and less likely to elicit an honest answer. Economic liberalism doesn't exist in a social and cultural vacuum; people half-consciously realize that economic liberalism works only if certain social and cultural preconditions are solidly in place, etc.
4. Since Pinker (2012) is a book, could you provide a page number reference?
5. "pertaining to" should be replaced with "about" throughout the text.
6."prohibitions against" and not "prohibitions over"
7. "which is a similar but somewhat construct"
This version is much better organized and I approve its publication.
A few minor criticisms :
1. The first hypothesis is clearly stated in the Introduction but the second hypothesis appears in a rather vague form:
original version: "There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis."
current version: "[][font=sans-serif]Cognitive ability appears to have a non-monotonic relation to measures of [][font=sans-serif]economically liberal beliefs ..."
[]Why did you change the second hypothesis?
[]2.[] "rate themselves on social liberalism and economic liberalism" can be changed to "rate their degree of adherence to social and economic liberalism"
[]3. I'm not sure why the correlation is weaker with social liberalism than with economic liberalism. This point should be addressed in the Discussion section. I can think of several possible reasons. Social liberalism has a weaker theoretical basis and may be perceived as less "scientific" than economic liberalism. Social issues tend to be more controversial and less likely to elicit an honest answer. Economic liberalism doesn't exist in a social and cultural vacuum; people half-consciously realize that economic liberalism works only if certain social and cultural preconditions are solidly in place, etc.
4. Since Pinker (2012) is a book, could you provide a page number reference?
5. "pertaining to" should be replaced with "about" throughout the text.
6."prohibitions against" and not "prohibitions over"
7. "which is a similar but somewhat construct"
Peter,
Thanks for looking it over.
I think the first file is written by me and the second is written mostly by Noah. He wrote the first draft, I added more stuff/edited it. So, minor changes in wording will probably be due to this change in authorship.
If you think it is problematic in this case, we can amend it.
I changed it to "[]As noted, in the original survey, respondents were asked to rate their own agreement with social liberalism and economic liberalism, respectively.". Think this is a bit better than both, and more in line with the actual given question on that survey.
[]
It's unwise to make much of such small differences. The sample is not large enough to say whether this is real or not. To be sure, I calculated the confidence intervals for the difference in correlations, both before and after adjusting for measurement error. Neither of these indicate that we can be sure that this difference is real. I added it to the supplementary materials section in the notebook.
http://rpubs.com/EmilOWK/208757 (look in the very bottom)
If you can find some funding, we could replicate this study with a larger sample and a better IQ test. Would cost 2-3k USD.
I only have an electronic version, which has no comparable page numbers, so I don't know which page it is on. However, it is in chapter 5 and I have added that.
Noah prefers the current phrasing sounds better. I don't care much, but favor "about". So I'll leave it as it is.
Fixed.
Fixed.
---
Updated the files.
Thanks for looking it over.
1. The first hypothesis is clearly stated in the Introduction but the second hypothesis appears in a rather vague form:
original version: "There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis."
current version: "[][font=sans-serif]Cognitive ability appears to have a non-monotonic relation to measures of [][font=sans-serif]economically liberal beliefs ..."
[]Why did you change the second hypothesis?
I think the first file is written by me and the second is written mostly by Noah. He wrote the first draft, I added more stuff/edited it. So, minor changes in wording will probably be due to this change in authorship.
If you think it is problematic in this case, we can amend it.
[]2.[] "rate themselves on social liberalism and economic liberalism" can be changed to "rate their degree of adherence to social and economic liberalism"
I changed it to "[]As noted, in the original survey, respondents were asked to rate their own agreement with social liberalism and economic liberalism, respectively.". Think this is a bit better than both, and more in line with the actual given question on that survey.
3. I'm not sure why the correlation is weaker with social liberalism than with economic liberalism. This point should be addressed in the Discussion section. I can think of several possible reasons. Social liberalism has a weaker theoretical basis and may be perceived as less "scientific" than economic liberalism. Social issues tend to be more controversial and less likely to elicit an honest answer. Economic liberalism doesn't exist in a social and cultural vacuum; people half-consciously realize that economic liberalism works only if certain social and cultural preconditions are solidly in place, etc.
[]
It's unwise to make much of such small differences. The sample is not large enough to say whether this is real or not. To be sure, I calculated the confidence intervals for the difference in correlations, both before and after adjusting for measurement error. Neither of these indicate that we can be sure that this difference is real. I added it to the supplementary materials section in the notebook.
http://rpubs.com/EmilOWK/208757 (look in the very bottom)
If you can find some funding, we could replicate this study with a larger sample and a better IQ test. Would cost 2-3k USD.
4. Since Pinker (2012) is a book, could you provide a page number reference?
I only have an electronic version, which has no comparable page numbers, so I don't know which page it is on. However, it is in chapter 5 and I have added that.
5. "pertaining to" should be replaced with "about" throughout the text.
Noah prefers the current phrasing sounds better. I don't care much, but favor "about". So I'll leave it as it is.
6."prohibitions against" and not "prohibitions over"
Fixed.
7. "which is a similar but somewhat construct"
Fixed.
---
Updated the files.
© Copyright 2024 OpenPsych - All Rights Reserved