[ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
2016-Jul-04, 09:30:39, (This post was last modified: 2016-Aug-04, 21:14:26 by Emil.)
#1
[ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
Journal:
Open Differential Psychology.

Authors:
Emil O. W. Kirkegaard
Julius D. Bjerrekær

Title:
Are stereotypes about immigrants accurate in Denmark?: a large, preregistered study

Abstract:
The study was preregistered with almost all analyses being specified before data collection began.

A nationally representative sample was asked to estimate the percentage of persons aged 30-39 receiving social benefits for 70 countries of origin (N = 766). After extensive quality control procedures, a sample of 484 persons were available for analysis. Stereotypes were scored by accuracy by comparing the estimates values to values obtained from an official source. Individual stereotypes were found to be moderately accurate (median/mean correlation with criterion values = .48/.43), while the aggregate stereotype was found to be very accuracy (r = .70). Both individual and aggregate-level stereotypes tended to underestimate the percentages of persons receiving social benefits and underestimate real group differences.

In bivariate analysis, stereotype correlational accuracy was found to be predicted by a variety of predictors at above chance levels, including conservatism (r = .13), nationalism (r = .11), some immigration critical beliefs/preferences, agreement with a few political parties, educational attainment (r = .20), being male (d = .19) and cognitive ability (r = .22). Agreement with most political parties, experience with ghettos, age, and policy positions on immigrant questions had little or no predictive validity.
In multivariate predictive analysis using OLS and LASSO regression, correlational accuracy was found to be predicted only by cognitive ability and educational attainment with even moderate level of reliability. In general, stereotype accuracy was not easy to predict, even using 24 predictors (k fold cross-validated R2 = 4%).

We examined whether stereotypes were more or less accurate for the more Muslim groups. Stereotypes were found to be less accurate in that they underestimated the percentages of persons from more Muslim groups receiving social benefits (mean estimation error for Muslim groups relative to overall elevation error = -8.09 %points).

Key words:
stereotypes, stereotype accuracy, Denmark, immigrants, social benefits, group differences, Muslims, replication, preregistered, open data

Length:
~7100 words, excluding references. PDF is 29 pages including references and appendix.

Files:
https://osf.io/wxqma/
Note that the questionnaire is currently not available. It will be made available shortly (pending email reply from the survey handler).

External reviewers:
Sean T. Stevens, who has published on stereotype research before and who reviewed the pilot study, has agreed to review this paper as well.
Reply
2016-Aug-07, 16:05:47,
#2
RE: [ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
The paper is interesting and is an example of carrying out a study plan with attention to details. My primary concern with the paper involves the nature of what is a stereotype and whether the study actually measured stereotype accuracy or instead measured aggregate guessing, possibly influenced by stereotypes. The relatively low correlations suggest the latter.

The usual usage of "stereotype" is in the context of social memes about groups. As we all know, these exist for most identifiable groups (races, nationalities, social groups, etc.) and have persisted for long periods; much of the comedy we enjoy is based on stereotypes. In this study, asking people about the amount of government support received by a group, falls into a category that I believe does not follow stereotyping because it is information that is not accurately known by almost the entire population of any country.

If people were asked to describe Muslims (because this group is central to the study), the descriptions are likely to involve stereotypes pertaining to appearance and behavior. I would expect this category of stereotypes to have a much higher accuracy than the items addressed by the paper.

As a matter of curiosity, I searched Google Scholar for "stereotype -threat" to exclude stereotype threat studies. The results returned a fairly large number of papers. Three that may be worth skimming (I don't have access to them):

http://jcr.oxfordjournals.org/content/21/2/354

http://jcr.oxfordjournals.org/content/21/2/354.abstract

http://psycnet.apa.org/psycinfo/2003-88371-012
---

Solutions

The issue described above may be resolved by a careful description of the objective of the study, noting that the questions being asked were intentionally selected to require respondents to call upon their images of groups (immigrants) to guess an answer. Or, the stereotype might be identified separately from the response, for example: the immigrant is associated with a prosperous or not prosperous group (stereotype) and this indicates the direction of guessing how much government assistance is received by the group. I think the second approach is better.

The authors could simply stick to the plan and explain that there are very large numbers of stereotypes and the ones they are examining come from a rarely probed category. This approach seems less attractive to me, but it might work with some extra discussion of objective, expectations (magnitude of correlation or accuracy measure), and the notion of strong versus weak stereotyping.

I looked at dictionary definitions of "stereotype" and found a wider range than I expected. One that I think works is from Dictionary.com: "a simplified and standardized conception or image invested with special meaning and held in common by members of a group." I think this definition works well for the things most people consider to be stereotypes.
---
"Participants were asked to rate themselves (0-100) on four scales: conservatism,
nationalism, economic liberalism and personal liberalism."

Were the definitions given to the participants? Were there instructions about how to choose from 0-100? Is the use of 100 points more meaningful than a narrower scale (5, 9, etc.)? I ask because a person is likely to select a number somewhat randomly and not feel that it is more or less accurate than a number over a range. I believe that people simply don't have very fine grained resolution of their reactions to questions.
---
"Participants were asked to rate their agreement (0-100) with each of the main political
parties in Denmark (15 parties), including parties outside parliament. They had the option
of stating that they had no preference or no knowledge about a particular party. This was
because the list included relatively unknown parties outside parliament."

Were party platforms given to participants? If not, are they understood by the range of people who are being asked? This item is somewhat difficult to understand from my perspective, because we have only two serious parties. I understand that more parties exist in other nations, but have no idea how they are understood by their citizens.

Text comments

These are suggestions for wording that may enhance the readability of the paper.

We examined whether stereotypes were more or less accurate for the more* Muslim groups. Stereotypes
were found to be less accurate in that they underestimated the percentages of persons from more**
Muslim groups receiving social benefits (mean estimation error for Muslim groups relative to overall
elevation error = -8.09 %points).

* change more to -- predominantly
** "more" is unclear here. Deleting the word seems to work, or another wording may clarify the meaning.
---
It found that stereotypes were moderately accurate (median correlational accuracy score = .51), but the
results are hard to generalize to the general* population.

* Following "generalize" the word "general" should be changed. Suggestions: national, or overall.
---
"We also noted that some users'
estimates were reverse of reality, indicating that they did not understand the task or that they were
purposefully filling it out in reversely." *

* suggest for last two words: "in reverse" or "backwards"
---
"Users that * fail these questions could then easily
be filtered afterwards. Additionally, we changed the order of the cognitive items to be random so that
the presenting order** could not have a systematic effect on the responses."

"For our analyses, we excluded all participants that * failed one of the first 7 controls or who gave
reversed answers"

* The use of "that" is not a serious error, but it is likely to be disconcerting to most readers. There are many grammatical discussions of "who" vs. "that" on the web. This one is worthwhile:

http://blog.apastyle.org/apastyle/2012/0...-that.html

** change "presenting order" to "order of presentation" or "presentation order"
---
"If people rely upon GDP per capita to
estimate immigrant performance, then their estimates will be highly correlated with these * which they
are (r = -.79)."

* Something is needed to set the last three words off as a phrase. A comma at the point of the * may work.
Reply
2016-Aug-07, 21:23:21,
#3
RE: [ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
Bob,

Thanks for the review. We will think about it and get back to you.
Reply
2016-Aug-16, 22:17:17,
#4
RE: [ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
This is far more informative than the average stereotype threat study. I was impressed by the care taken to clean the data and filter out noncompliant respondents. My only suggestion is that the term stereotype should be briefly defined in the introduction.
Reply
2016-Aug-16, 23:28:53,
#5
RE: [ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
Gerhard,

Thank you fro taking the time to read it over. It is pretty long. We spent a large number of hours in discussing in emails with the pollster trying to filter out the noncompliants. We will probably be returning to this issue as it is somewhat important for interpretation of these kinds of studies.

We will expand the introduction to better deal with the term "stereotype" which was also a suggestion by Bob Williams (above).
Reply
2016-Aug-17, 02:54:43, (This post was last modified: 2016-Aug-17, 02:55:10 by Emil.)
#6
Updated version for Bob and Gerhard
Bob,

Quote:The paper is interesting and is an example of carrying out a study plan with attention to details. My primary concern with the paper involves the nature of what is a stereotype and whether the study actually measured stereotype accuracy or instead measured aggregate guessing, possibly influenced by stereotypes. The relatively low correlations suggest the latter.

There are two parts to this: semantic/conceptual and empirical.

Semantic. We use the word stereotype in the meaning used by many other researchers and recommended by Lee Jussim, the current foremost expert on stereotypes in my opinion (One the papers you link to is in fact an old review by Lee Jussim!). In this context, a stereotype is a belief about a group. The beliefs may be and are mostly statistical in nature, not absolute (many/few not all/no). Some have used the word in other senses usually involving defining stereotypes as inaccurate or exaggerated beliefs about groups (you give an example of this in your post). However, this leaves us with the problem of what to call beliefs about groups that are accurate (such as beliefs about the clothing that you mention), and also results in various logical problems that are dealt with at length in Jussim's 2012 book. I recommend reading the book. The book can be freely downloaded on Libgen (http://gen.lib.rus.ec/book/index.php?md5...885DFB25CA ).

The use of stereotype this way is also in line with the popular stereotype threat hypothesis (mentioned by you) which is that making stereotypes about lower performance about a group salient depresses that group's performance via some threat response/emotional stress. The most two commonly researched cases are the lower scholastic/cognitive ability of African Americans and the lower math performance by girls. The stereotypes being made salient if of course the common beliefs that African Americans and females perform worse on these tasks.

Per request (you and Gerhard), I have added a few lines explaining the use of the word to readers who are unfamiliar with this use.

Empirical. The stereotype accuracy correlations reported in this study are 1) strong compared to most social science effects, especially social psychology (http://neuron4.psych.ubc.ca/~schaller/Ps...ta2003.pdf ), and 2) very similar to other studies (see various reviews by Jussim).


Quote:"Participants were asked to rate themselves (0-100) on four scales: conservatism, nationalism, economic liberalism and personal liberalism."

Were the definitions given to the participants? Were there instructions about how to choose from 0-100?

Yes and yes. You can verify this by reading the questionnaire in the project files.

Quote:Is the use of 100 points more meaningful than a narrower scale (5, 9, etc.)? I ask because a person is likely to select a number somewhat randomly and not feel that it is more or less accurate than a number over a range. I believe that people simply don't have very fine grained resolution of their reactions to questions.

The use of a scale with more options reduces the impact of the violation of statistical assumptions, i.e. that of continuity. I will be exploring the effect of using a 7-level (1-7) scale vs. a 101-level (0-100) scale in an upcoming study (with Noah Carl). It's an empirical question of how fine-grained people's self-ratings are if one gives them the chance. As far as I know, no one knows because I looked but could not find a study exploring this matter. However, one finding from the Good Judgment Project was that better forecasters used more fine-grained predictions. You can read about this in Tetlock's book Superforecasting: The Art and Science of Prediction. https://en.wikipedia.org/wiki/Superforecasting

Quote:"Participants were asked to rate their agreement (0-100) with each of the main political parties in Denmark (15 parties), including parties outside parliament. They had the option of stating that they had no preference or no knowledge about a particular party. This was because the list included relatively unknown parties outside parliament."

Were party platforms given to participants? If not, are they understood by the range of people who are being asked? This item is somewhat difficult to understand from my perspective, because we have only two serious parties. I understand that more parties exist in other nations, but have no idea how they are understood by their citizens.

No and yes. All the main parties are widely known to the public as well as their approximate positions on policy matters. I suggest you spend some time reading about multi-party systems on Wikipedia. The US is the outlier here, all other Western countries have more than two parties, even the UK which uses a similar system (first past the post).

-

Quote:We examined whether stereotypes were more or less accurate for the more* Muslim groups. Stereotypes were found to be less accurate in that they underestimated the percentages of persons from more** Muslim groups receiving social benefits (mean estimation error for Muslim groups relative to overall elevation error = -8.09 %points).

* change more to -- predominantly
** "more" is unclear here. Deleting the word seems to work, or another wording may clarify the meaning.

The use of gradual words is intentional. Being Muslim is not dichotomous for groups. For instance, Nigerians are about 50% Muslim, 50% Christian. Take a look at Pew Research's work on the topic which is where the Islam data comes from. http://www.pewforum.org/2011/01/27/table...y-country/

Quote:It found that stereotypes were moderately accurate (median correlational accuracy score = .51), but the results are hard to generalize to the general* population.

* Following "generalize" the word "general" should be changed. Suggestions: national, or overall.

Changed to overall.

Quote:"We also noted that some users' estimates were reverse of reality, indicating that they did not understand the task or that they were purposefully filling it out in reversely." *

* suggest for last two words: "in reverse" or "backwards"

Changed to We also noted that some users' estimates were opposite of reality, indicating that they did not understand the task or that they were purposefully answering dishonestly.

Quote:"Users that * fail these questions could then easily be filtered afterwards. Additionally, we changed the order of the cognitive items to be random so that the presenting order** could not have a systematic effect on the responses."

"For our analyses, we excluded all participants that * failed one of the first 7 controls or who gave reversed answers"

* The use of "that" is not a serious error, but it is likely to be disconcerting to most readers. There are many grammatical discussions of "who" vs. "that" on the web. This one is worthwhile:

http://blog.apastyle.org/apastyle/2012/0...-that.html

** change "presenting order" to "order of presentation" or "presentation order"
Changed to who and to order of presentation.

Quote:"If people rely upon GDP per capita to estimate immigrant performance, then their estimates will be highly correlated with these * which they are (r = -.79)."

* Something is needed to set the last three words off as a phrase. A comma at the point of the * may work.

Added a comma.

---

Updated the files to version 13 (paper.pdf and paper.odt).
Reply
2016-Aug-17, 22:19:02,
#7
RE: [ODP] Are stereotypes about
(2016-Aug-17, 02:54:43)Emil Wrote: -
Quote:We examined whether stereotypes were more or less accurate for the more* Muslim groups. Stereotypes were found to be less accurate in that they underestimated the percentages of persons from more** Muslim groups receiving social benefits (mean estimation error for Muslim groups relative to overall elevation error = -8.09 %points).

* change more to -- predominantly
** "more" is unclear here. Deleting the word seems to work, or another wording may clarify the meaning.

The use of gradual words is intentional. Being Muslim is not dichotomous for groups. For instance, Nigerians are about 50% Muslim, 50% Christian. Take a look at Pew Research's work on the topic which is where the Islam data comes from. http://www.pewforum.org/2011/01/27/table...y-country/

The problem remains that the wording used above will not be clear to many readers. The intent could be explained, using parenthesis or footnote, or worded so that the intent is clear. A reader could translate "more Muslim groups" as meaning:

►a larger number of distinct Muslim groups
►more intense or observant Muslim groups
►additional Muslim groups

Other than this small item, I think you have addressed the things I mentioned and think the paper is well written and interesting.
Reply
2016-Aug-17, 22:58:06,
#8
RE: [ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
Bob,

The context seems to rule out your (1) and (3) interpretations, e.g. context for the first use rules our (1) due to the definite article. The second one rules out (3). (2) is possible, however.

I have changed the wording to a longer less ambiguous phrasing "groups with larger proportions of Muslims" and variants. Also rephrased the abstract slightly.

Files updated (version 14).
Reply
2016-Aug-18, 01:32:36, (This post was last modified: 2016-Aug-18, 13:44:07 by BobWilliams.)
#9
RE: [ODP] Are stereotypes about
Emil - Thanks. I think the sentence is clear now. In fact, I think I was guessing incorrectly on all three of my interpretations.

I see nothing else to change. I approve for publication.
Reply
2016-Aug-20, 17:30:32,
#10
[ODP] Are stereotypes about immigrants accurate in Denmark?: a large, preregistered
This is a lege artis conducted study and a very well prepared paper.

Some points can be improved in the manuscript and some aspects in further studies.

First, there is nearly no introduction and no theory. Its a very Ockham-British-Lynn like data driven paper. More verbalization and generally a better formatting of ODP papers would be helpful.

Next there is a lot of statistical analysis and it seems the authors have analyzed all what is possible to do with the given data. However, data analysis should be driven by research questions and theory.

Introduction: You wrote “stereotypes were moderately accurate (median correlational accuracy score = .51)”. Usually, taken Cohen’s levels for interpretation r=.10 is a small correlation, .30 a medium and .50 a large correlation. For the validity of tests r=.50 is taken as a large correlation (Fisseni).
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum.
Fisseni, H.-J. (2004/1997/1990). Lehrbuch der psychologischen Diagnostik. Göttingen: Verlag für Psychologie.
However, for reliability measures higher correlations are expected (r=.80 medium, r=.90 high; Fisseni). And as I remember averaging Jussim’s (2012, his tables 17-1 to 17-3) results the average correlation between stereotypes and criteria is about r=.81.
So you should discuss possible benchmarks/norms and justify your taken benchmark for interpretation.

I see two aspects that could be improved in the survey hopefully possible in a future/further study:

1. Disclosure. You start your questionnaire (translated to English) with:
“When people have expectations about how one or more groups are or behaves, it’s often called a “prejudice” or “stereotype”. Often it’s said that they are inaccurate, even without examining this. The purpose of this study is to examine how precise the Danes’ stereotypes of immigrants are. Which is why we ask of you to evaluate how well a plethora of immigrant groups in Denmark perform, without looking up the numbers beforehand.”
In this way the surveyed persons know what is going on and that may bias their answers and your results. You should better frame it more neutrally and bend thoughts [distract] from the stereotype subject.

2. Ask more questions. Not only ask about migrants and public assistance (welfare), but also about crime and about divorce and family stability or production of jobs (anything positive on immigrants) or student performance. And a totally different subject as differences between men and women.

In the paper itself the question has to be mentioned, this is central! Currently the paper cannot be understood. (“People who live in Denmark originate from many countries. We would like you to evaluate how many people among the 30-39 year olds that you think are on public assistance, from each country of origin.”)

Very, very good: Your control questions and the procedure to clean the data. This is exemplary!

Always add page numbers in manuscripts. Always add page numbers in by Open Psych published papers. Papers should be finally formatted as usual by professional publishers (Cambridge, Elsevier, Sage, Springer). The more professional the better.

I do not understand the utility of the analyses in chapter 6 (“Inter-rater agreement”). Start your analyses and paper with theory and research questions and then add only the necessary analyses.

Very good and important: What predicts stereotype accuracy. Here you found that conservatism leads to higher accuracy of judgments. However, in areas of lefty questions (e.g. gender differences) maybe progressivity would lead to better stereotype accuracy.

As far as I remember also Lee Jussim has done some analyses on predictors/determinants of stereotype accuracy. Compare your results with his results and bring it in the introduction and discussion.

Table 5 in chapter 7.1: Never bring abbreviations – I do not understand what means “DF” etc.

Below Figure 6 (why there are no page numbers?): d=.19 has to be written as d=0.19. Values (p, r, usually beta) which can be only between 1, 0 and –1 write as “.19”, values that can be larger (d) write as “0.19”.

Finally, choose different outlets. Also include standard APS, APA and Elsevier journals.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)