Hello There, Guest!  

[OQSPS] Cognitive ability and political preferences in Denmark

#11
Bob,

Thanks for looking over the study.

Quote:In the abstract there is a general description of what was done, but no clear statement of objective. I think most readers would want to see something that clearly states something about the purpose of the study.

I have rewritten the abstract to briefly note why we carried out the study. I added: Multiple studies have reported positive relationships between cognitive ability and preferences for freedom, both at the personal level (e.g. drug use) and the economic (e.g. smaller government). To add to this, we investigated the ...


Quote:I am confused by this:
By going through a pollster, we administered 24 additional items to the sample:
After reading again, I assume that this simply means that when you used the pollster, you added the additional items. If so, the confusion I experienced was related to the word "by." It could be written to say "When the polling was repeated with a pollster, we added 24 additional items." ... Or something to that effect.


The final sample size was n = 327 (65% of the original n).
On first reading, it was unclear as to why the final sample size was smaller. I now assume that 327 was the number of people who responded to the pollsters, although it was not obvious to me that they contacted all of the first group or only part. Can this be worded differently?
Later (section 2.3) there is a comment about removing responses where random answering was indicated, but that resulted in a 20% reduction. Was the 20% reduction before or after the initial reduction? I assume that is what happened.

I have rewritten the text in multiple places to be more clear.

What was done was that:

An earlier study surveyed some n=552 Danes on various variables for the stereotype study (). We arranged with the pollster to send out invitations to take part in a second survey to the same 550 persons. Some of them had either quit taking part in surveys all together or simply did not want or have time to take part in our second survey. For this reason, the sample size is smaller than the one before at n=333. Some people failed one or (usually) more of the control questions and due to an error at the pollster, some could not be linked up between the two surveys. This left us with n=259 persons with complete data for the analyses.



Quote:As Table indicates, responders were slightly younger, had slightly lower cognitive ability and were slightly less educated than non-responders. There was, therefore, some selection bias in responding to our survey.

Some readers will recall that the usual pattern of participation is skewed towards brighter people, due to their greater willingness to participate. This case went the other way. I assume this is random, but it might be worth a sentence or two to explain the likely cause of this somewhat unexpected response makeup.

I considered it, but since the difference is small (d=23), the reversed finding probably reflects a chance happening and is not worth commenting on.


Quote:Just a comment on the Likert scales: It seems to me that by dividing the scales into two ranges, an unnecessary distraction resulted. This meant lots of discussion about combining the results when there was nothing gained with respect to the paper. Using either one would have, in my opinion, been better and would not have resulted in any loss of value. Given that this was already done, there is no way to change it now.

I agree, but we didn't know that before carrying out the study! :) Now we know that it doesn't matter whether one uses 7-point or 101-point scales.


Quote:Figures 3 and 4
Some of the item characteristic curves seem to have low slopes, making the inflection point indistinct. I understand that you can define the latent trait despite this, but it appears to me that those curves are not what you would want. The factor analysis shows 4 distributions that show reasonable looking shapes, but the others seem to be uninformative. There was little discussion of this, but there was a mention of too few easy items. Can more be said about the impact of this skew on the outcome? It leaves me wondering how one can justify using uneven item difficulty to indicate the relations that are discusses with respect to population preferences versus intelligence.


The figures (one was missing and has been added) are primarily for the readers more expert in IRT. However, I have rewritten some of the text to be more informative. The missing figure was the one showing the sum of the information from the items, which makes it clear that there was a lack of discriminative ability on the left tail. But in general, it was not that large a problem: mean pass rate was 37% with the very difficult item and 41% without which is pretty close to the optimal value of 50%.

I realize that the IRT figures are ugly. However, it is not easy to produce better ones because the figures are generated by the package that calculates the IRT results (psych) and they are based on the archaic base plotting system, not the modern ggplot2 system.


Quote:The above may be related to my comments concerning figures 3 and 4. The papers I have read on the subject of political orientation on left-right scale or liberal-conservative scales have shown that there is a correlation with intelligence (as you acknowledge near the end of the paper), although the results have been somewhat mixed. So, I too am surprised by this result and ask if it can be traced to the failure to use a population representative measurement of intelligence. I am a bit more confused in that the ICAR5 should have given an adequate measurement. I admit to confusion, which may be entirely mine and not a problem with the methodology.


The somewhat mixed results comes from some researchers trying to model political preferences with only 1 dimension (liberal-conservative). The mixed results to some degree reflect the different ways different researchers have operationalized this supposed 1-dimensional trait. If they used many items related to 'social conservatism' (e.g. opposition to legalized abortion) they obtain negative correlations, but if they use fiscal/economic measures, the relationships turn positive. If they used some kind of mix, the results will be unclear.

--

As it happens, there had been a bug in the IRT software (scoreIrt(), https://cran.r-project.org/web/packages/psych/news.html). The bug meant that the scores from IRT were incorrect and was especially problematic for short scales (such as ours), which affected the cognitive ability estimates from this study. The bug has now been corrected and this increased the correlations to the point that they are more in line with the theoretical expectations, especially after correction for measurement error. In fact, most of the tables and figures had to be replaced because of this error.

This also meant that much of the discussion had to be rewritten which I have done. I have also inserted proper tables in the paper as well and prettier and better labelled figures.

I have updated the files.

 Reply
#12
Emil - The changes you made seem to correct the points that were unclear. Some of the changes were a great improvement. I will download and read the revised text. I assume it will be good and, if so, I will approve.
 Reply
#13
I approve the paper for publication.  I would like to offer one additional suggestion:

which meant that it provided less discriminability in the left-hand tail.

The word "discriminability" is awkward.  Perhaps you could find alternate words.  For example... "it provided lower resolution in the left-hand tail." Or "it was less sensitive to detecting differences in..."  Or "it did not provide the desired level of discrimination in.."
 Reply
#14
Given the current social attention to various forms of undesired or allegedly undesired forms of discrimination, the word is somewhat odd to use. However, this is the standard term used in IRT and we merely follow normal practice. See e.g. https://www.goodreads.com/book/show/1107...nse-theory

I quote from the introduction:


Quote:Item Discrimination
A higher discrimination means that the item differentiates (discriminates) between examinees with different levels of the construct. Thus, high discrimination is desirable. The purpose of using the instrument is to differentiate (discriminate) between examinees who know the material tested and those who do not, or on an attitude scale, between those who have positive attitudes and those who have negative attitudes. In CTT, the corrected item-total point-biserial correlation is the typical index of discrimination; when this is positive, examinees who answer the item correctly (or endorse the item) score higher on the sum of the remaining items than do those who answer the item incorrectly (or disagree with the item). In IRT, an index symbolized as a is a measure of the item discrimination. This index is sometimes called the slope, because it indicates how steeply the probability of correct response changes as the proficiency or trait increases. In both CTT and IRT, higher values indicate greater discrimination.


Now, we used the variant discriminability to avoid the precise version of the word. This is also done by many others as a quick search will show.
 Reply
#15
Quote:Now, we used the variant discriminability to avoid the precise version of the word. This is also done by many others as a quick search will show.

Okay, that is fine with me.  It was only a comment.
 Reply
#16
This paper could be better organized. The Introduction should be longer with an explanation of the enigmatic sentence "Many studies have examined the relationship between cognitive ability and political preferences." At the very least, you should provide some references for that sentence. Who else has studied this relationship? What did they find? At one point, you mention Steven Pinker. Has he written on this specific subject?

The working hypotheses should be stated in the Introduction, and not further in the text. These are in fact stated on p. 22:

- Cognitive ability will be positively related to the freedom end of both axes.
- There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis.

These are interesting hypotheses. What is the rationale for them? Are they based on previous findings in the literature? Are they a personal hunch? Please tell us.


The Results section should be shorter. Much of the data analysis could be summarized or placed in an appendix. The conclusions should likewise be summarized and not provided piecemeal. The main conclusion seems to appear on p. 45: "In general, the correlations are near zero as expected, but there are a few exceptions. Smarter people seem to be somewhat against the government unsuring jobs [reducing job security?] higher minimum wages, and somewhat for blasphemy and flag burning being legal."

There should be a Discussion section. Why are the correlations so low? How does this finding compare with other findings in the literature?
 Reply
#17
Thanks Peter.

We'll work on a revision.
 Reply
#18
Peter,

Did you perhaps read an old version of the submission? The one I'm looking at does not seem to match the one you comment about. E.g. there is no "many studies have" phrase.

Maybe I made a mistake. I have re-uploaded the files. It's the paper.docx/paper.pdf files.

https://osf.io/xdpcq/files/
 Reply
#19
Yes, I read doc.docx instead of paper.docx.

This version is much better organized and I approve its publication.

 A few minor criticisms :

1. The first hypothesis is clearly stated in the Introduction but the second hypothesis appears in a rather vague form:

  original version: "There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis."
  current version: "Cognitive ability appears to have a non-monotonic relation to measures of economically liberal beliefs ..."

Why did you change the second hypothesis?

2. "rate themselves on social liberalism and economic liberalism" can be changed to "rate their degree of adherence to social and economic liberalism"

3. I'm not sure why the correlation is weaker with social liberalism than with economic liberalism. This point should be addressed in the Discussion section. I can think of several possible reasons. Social liberalism has a weaker theoretical basis and may be perceived as less "scientific" than economic liberalism. Social issues tend to be more controversial and less likely to elicit an honest answer. Economic liberalism doesn't exist in a social and cultural vacuum; people half-consciously realize that economic liberalism works only if certain social and cultural preconditions are solidly in place, etc.



4. Since Pinker (2012) is a book, could you provide a page number reference?

5. "pertaining to" should be replaced with "about" throughout the text.

6."prohibitions against" and not "prohibitions over"

7. "which is a similar but somewhat construct"
 Reply
#20
Peter,

Thanks for looking it over.


Quote:1. The first hypothesis is clearly stated in the Introduction but the second hypothesis appears in a rather vague form:

  original version: "There will be a non-linear relationship between cognitive ability and the economic liberalism scale such that the positive slope will decline in strength or perhaps reverse near the end of the ability axis."
  current version: "Cognitive ability appears to have a non-monotonic relation to measures of economically liberal beliefs ..."

Why did you change the second hypothesis?




I think the first file is written by me and the second is written mostly by Noah. He wrote the first draft, I added more stuff/edited it. So, minor changes in wording will probably be due to this change in authorship.

If you think it is problematic in this case, we can amend it.


Quote:2. "rate themselves on social liberalism and economic liberalism" can be changed to "rate their degree of adherence to social and economic liberalism"


I changed it to "As noted, in the original survey, respondents were asked to rate their own agreement with social liberalism and economic liberalism, respectively.". Think this is a bit better than both, and more in line with the actual given question on that survey.


Quote:3. I'm not sure why the correlation is weaker with social liberalism than with economic liberalism. This point should be addressed in the Discussion section. I can think of several possible reasons. Social liberalism has a weaker theoretical basis and may be perceived as less "scientific" than economic liberalism. Social issues tend to be more controversial and less likely to elicit an honest answer. Economic liberalism doesn't exist in a social and cultural vacuum; people half-consciously realize that economic liberalism works only if certain social and cultural preconditions are solidly in place, etc.


It's unwise to make much of such small differences. The sample is not large enough to say whether this is real or not. To be sure, I calculated the confidence intervals for the difference in correlations, both before and after adjusting for measurement error. Neither of these indicate that we can be sure that this difference is real. I added it to the supplementary materials section in the notebook.

http://rpubs.com/EmilOWK/208757 (look in the very bottom)

If you can find some funding, we could replicate this study with a larger sample and a better IQ test. Would cost 2-3k USD.


Quote:4. Since Pinker (2012) is a book, could you provide a page number reference?

I only have an electronic version, which has no comparable page numbers, so I don't know which page it is on. However, it is in chapter 5 and I have added that.


Quote: 5. "pertaining to" should be replaced with "about" throughout the text.


Noah prefers the current phrasing sounds better. I don't care much, but favor "about". So I'll leave it as it is.


Quote:6."prohibitions against" and not "prohibitions over"


Fixed.

Quote: 7. "which is a similar but somewhat construct"


Fixed.


---

Updated the files.
 Reply
 
Forum Jump:

Users browsing this thread: 1 Guest(s)