Back to [Archive] Post-review discussions

[ODP] Immigrant GPA in Danish primary school is predictable from country-level variab
Laird has been so kind as to proofread it. The latest draft (#5) is on OSF now along with the new source code etc.

https://osf.io/p9d5z/

The new draft has a new section re. the size of the GPA gaps, which includes both calculations on a country-level and at the total immigrant population level by generation.


There are some language issues. For example:

"A large body of research shows that immigrant performance and traits are predictable from their or their parents’ countries of origin."

What you mean is e.g.,

"A large body of research shows that the traits of immigrant groups can often be predicted from the traits of the inhabitants of their (or their ancestors') country or region of origin."

"Since g is not the only factor that causes differences in g, one would not expect a g difference of 1.0 d to be associated with a 1.0 d difference in GPA. I could not find a study that reports the correlation of IQ with GPA in the final exams in Denmark, but such a correlation would likely be around .5-.7. For the UK, the correlation of g/IQ with the similar GSCE exams has been reported as .58 and .69.[23, 24]"

In the US, GPA is grade point average which is based on more than exams; it is also based on homework and class participation. If Danish GPA is a similar construct, the correlation between it and IQ should be lower than what you suggest (at around 0.3 to 0.5, see appendix A here). Is Danish GPA like UK standardized tests (GCSE exams) or like US grade point averages? Please clarify in the paper.

What's up with footnote (3)?

Would you mind rerunning the correlations with native Danes excluded and letting me know if the results are similar. Immigrants (i.e., 1st and 2nd gen) often systematically perform differently than natives (i.e., a true immigrant effect), so there can be a immigrant x origin effect. I have found one before.
Admin
John,

I replaced my version with yours.

---

Good question.

The first report with data writes:

Analysen baserer sig på detaljerede data på individniveau, indhentet fra Danmarks Statistik. Dette inkluderer oplysninger om de karakterer, hver
enkelt folkeskoleelev har opnået ved afgangseksamen i 9. klasse i perioden fra 2007 til 2009. Undersøgelsen er afgrænset til at omfatte resultater fra prøverne i de bundne eksaminer med ekstern censur ud fra en vurdering af, at karaktererne i disse eksamensprøver i højere grad kan sammenlignes end de, hvor eksterne bedømmere ikke har medvirket.

Følgende prøvekarakterer er dermed inddraget i undersøgelsen:
• Dansk læsning
• Mundtlig dansk
• Dansk orden
• Dansk retskrivning
• Dansk skriftlig fremstilling
• Mundtlig engelsk
• Mundtlig fysik/kemi
• Matematiske færdigheder (skriftlig)
• Matematisk problemløsning (skriftlig)


In my translation:

The analysis is based on detailed individual-level data which is retrieved from Statistics Denmark. This includes information about the grades every public school student has received at the final exams in the 9th grade in the period 2007 to 2009.
The analysis is restricted to the results from the mandatory exams with external censorship because these were judged to be easier to compare than those where external censors did not take part.

The following exam grades are thus included in the study:
- Danish reading
- Verbal Danish
- Danish order
- Danish spelling
- Danish writing
- Verbal English
- Verbal physics/chemistry
- Mathematical proficiency (written)
- Mathematical problem solving (written)


The second report writes:

De karakterer, som der ses på, er karakterer i de såkaldt bundne prøvefag ved folkeskolens afgangsprøve. Det drejer sig om:

Dansk, mundtlig
Dansk, læsning
Dansk, retskrivning
Dansk, skriftlig
Matematik, matematiske færdigheder
Matematik, matematisk problemløsning
Engelsk, mundtlig
Fysik/kemi, praktisk/mundtlig

Karakterer i alle de ovenstående fag er lagt sammen ved udregningen af de samlede gennemsnit for alle bundne prøvefag. I de fleste tilfælde er de seneste fem skoleår lagt sammen, så det er muligt at se på alle herkomstgrupper og underopdele nogle af grupperne på enkelte oprindelseslande. Dansk, orden har i nogle af årene også været et bundet prøvefag, men det er ekskluderet i alle beregninger. Elever i specialklasser og privatister (elever som ikke har mulighed for at aflægge prøve på en skole fx på grund af hjemmeundervisning) er heller ikke medtaget.


In my translation:

The grades we look at here are grades from the so-called mandatory exam classes at the public school's final exams. It concerns:

Danish, verbal
Danish, reading
Danish, spelling
Danish, written
Mathematics, proficiency
Mathematics, problem solving
English, verbal
Physics/Chemistry, practical/verbal

Grades in the above mentioned classes are summed in the calculation of the overall average for all mandatory exam classes. In most cases the five years [where data are from] are treated as one, so as to make it possible to examine all origin groups and further divide some of the groups into single origin countries. Danish, order has in some of the years also been a mandatory exam class, but it is excluded from all calculations. Students in special education classes and privatists (students who don't have the option to take the exam at a school e.g. because of home education) are not included either.


So, these are only the finals exams, not grades from the school year which tend to be mixed with non-g factors e.g. desire to please the teacher. I added in the paper:

The GPAs from the reports are based only on test results, not yearly grades, so they are less influnced by things such as teacher-student relationship. See the peer review discussion for details.

The classes are the same except that order was included in the first and excluded in the second.

It is also noteworthy that students in special ed. are excluded. There are more immigrants in these classes per capita (at least, that has been claimed in the press), so this would tend to reduce the GPA gap a bit.

I decided to look up the numbers in UNI-C. I found some numbers by generation. They are here in sheet #3.

Immigrants are overrepresented in special education classes, but only the second generation. They are also over-represented in the communal youth schools, which are often attended by children who have problems.

On the other hand, they are under-represented in the continuation schools with special offers. According to here, it is a holistic offer for students with special needs such as dyslexia or "general learning problems".

Now, if immigrants have lower GI in Denmark, one would certainly expect more marked numbers than these assuming that there is no immigrant status x diagnostization effect.

---

What's up with footnote (3)?


Nothing. It is a single word that is a hyperlink to the URL in question. I have moved it into the text to avoid unnecessary creation of a footnote.

---

Would you mind rerunning the correlations with native Danes excluded and letting me know if the results are similar. Immigrants (i.e., 1st and 2nd gen) often systematically perform differently than natives (i.e., a true immigrant effect), so there can be a immigrant x origin effect. I have found one before.


Sure.

Results here in sheet #4. I also put both the Pearson and Spearman correlations there, and a table of their discrepancies. The average difference is very small, Pearson's r being .04 and .03 higher than Spearman's r.

Below, there is the With minus Without Denmark discrepancies. The results were generally weaker without Denmark, especially in the small dataset.

It is easier to see why if one looks at the scatterplots. These are attached.

Let me know what you want me to note about the result of excluding Denmark. Note that in small datasets, removing a few datapoints can drastically change results.
Below, there is the With minus Without Denmark discrepancies. The results were generally weaker without Denmark, especially in the small dataset. It is easier to see why if one looks at the scatterplots. These are attached. Let me know what you want me to note about the result of excluding Denmark. Note that in small datasets, removing a few datapoints can drastically change results.


I ran into this problem in a different analysis. What you should do depends on how you discuss and report the data. Since you report all of the relevant data, it's not a big issue. If you include Danes, you should describe the data as second generation +or second generation with Danes includes. I would tend not to include them since I'm generally interested in the relative performance of immigrant ethnic groups in a specific country not of all ethnic groups.
Admin
https://osf.io/p9d5z/

This one has:
  • Removed the analysis of restriction of range. This was technically flawed and not very informative due to very small samples.
  • Proofreading by a native
  • Analysis of combined GPA scores, new section
  • Analysis of data without Denmark, as requested by Fuerst /w scatterplots.
  • More data tables with Spearman's cors and discrepancy scores


I changed the submission to Research Article, since it is by now 11 pages, not a small thing anymore. It seems that any paper I try to publish here ends up being longer than I initially wanted due to reviewer requests.
https://osf.io/p9d5z/

This one has:
  • Removed the analysis of restriction of range. This was technically flawed and not very informative due to very small samples.
  • Proofreading by a native
  • Analysis of combined GPA scores, new section
  • Analysis of data without Denmark, as requested by Fuerst /w scatterplots.
  • More data tables with Spearman's cors and discrepancy scores


I changed the submission to Research Article, since it is by now 11 pages, not a small thing anymore. It seems that any paper I try to publish here ends up being longer than I initially wanted due to reviewer requests.


I find this version of the paper acceptable for publication.

In the future, it would be nice if the author's papers could cover more terrain. Perhaps he could combine and succinctly discuss several such analysis from different countries. Generally, it is tedious to read through numerous papers that, individually, marginally contribute to the understanding of the topic. I appreciate that others may feel quite differently.
Admin
The problem is that when papers get too long, I find it difficult to finish them up, and the review process takes longer.
Emil,

I have only a few minor criticisms and observations:

1. You should add a paragraph that would summarize the excellent points you made about immigrant overrepresentation in special education classes and communal youth schools. Truancy is another factor that might make IQ differences seem smaller than they really are.

2. "I chose Islam prevalence because of frequent media attention being given to this factor." Could you elaborate a bit? Do the media frequently discuss poor academic performance among the children of Muslim immigrants? Since you seem to attach a lot of importance to this variable, it wouldn't hurt to summarize what is said in the media and include a few quotes.

2. In your paper, the word "data" is treated sometimes as a plural noun and sometimes as a singular noun. Most style books recommend treating it as a plural noun (the singular form would be "datum"). So you should write "The data for the year 2013/2014 are here" and "GPA data are also available ...".

I like this paper and am more than willing to give my approval if you address the above points.
Admin
Thank you for the review, Peter.

Emil,

I have only a few minor criticisms and observations:

1. You should add a paragraph that would summarize the excellent points you made about immigrant overrepresentation in special education classes and communal youth schools. Truancy is another factor that might make IQ differences seem smaller than they really are.


I have added a paragraph to the discussion:

A problem with interpreting the results is that it only concerns normal schools. Special education schools are thus excluded, and they are disproportionately attended by lower GI students. Thus one would expect immigrants to be over-represented in special education classes and this would have the effect of reducing the GPA gap found in the normal schools. Immigrants are in fact over-represented in some forms of special education schools, especially the second generation (see the appendix for details). However, here we are assuming that there is no under-diagnostization of special educational needs among immigrant students. Such under-diagnostization is likely because social workers would try to avoid over-representation of immigrant students in schools for the retarded. It is also expected because a higher proportion of students below the cut-off (usually IQ 75) from a lower GI population would be socially normal because they are part of the left tail of their distribution as opposed to having a major genetic defect such as Down's Syndrome which makes it very plain that one has special educational needs.\cite{jensen1969much} It is currently unknown how large these biases are for the Danish data, so some caution is advised.


2. "I chose Islam prevalence because of frequent media attention being given to this factor." Could you elaborate a bit? Do the media frequently discuss poor academic performance among the children of Muslim immigrants? Since you seem to attach a lot of importance to this variable, it wouldn't hurt to summarize what is said in the media and include a few quotes.


I have added:

I chose Islam prevalence because of frequent media attention being given to this factor and because previous research has shown that it has incremental validity over cognitive measures and is a plausible environmental cause.\cite{kirkegaardfuerst2014} For instance, the conservative web-tabloid \textit{Den Korte Avis} (The Short Newspaper) criticized researchers from a public researcher institute for not taking culture into account\cite{DenKorteAvis}:

\begin{quote}
The research thus speaks only about breaking the negative social inheritance as early as possible, but does not mention the cultural inheritance and the creation of parallel societies as a substantial factor, and a focus area for combating the problems [with low performance of immigrant students in schools]. With this narrow, politically correct point of view one cannot straighten out the problems.
\end{quote}


2. In your paper, the word "data" is treated sometimes as a plural noun and sometimes as a singular noun. Most style books recommend treating it as a plural noun (the singular form would be "datum"). So you should write "The data for the year 2013/2014 are here" and "GPA data are also available ...".


I went thru the paper and hopefully changed all the instances to the plural. I don't mind which one is used, but consistency is preferable.

I made no other changes. New version uploaded. https://osf.io/p9d5z/
Emil,

1. I rewrote the paragraph to emphasize the point that the two tendencies would cancel each other out. Feel free to accept or reject what I wrote:

This study only concerns normal schools and excludes special education schools, which have a higher proportion of lower GI students. Immigrants may thus be over-represented in such schools, thereby reducing the GPA gap in normal schools and making it seem smaller than it really is for the Danish population as a whole. Immigrants are in fact over-represented in some special education schools, especially the second generation (see the appendix for details). On the other hand, there may be under-diagnosis of special education needs among immigrant students. Such under-diagnosis would happen partly because social workers may try to avoid assigning immigrant students to schools for the retarded and partly because a higher proportion of immigrants appear behaviorally normal even in the "retarded" range (usually less than an IQ of 75). In a population with a lower mean IQ, it is less likely that a below-75 IQ would have unusual genetic or developmental causes (e.g., Down's Syndrome) that disrupt many aspects of mental functioning. Low IQ individuals would thus seem more behaviorally normal and be less likely assigned to special education classes.\cite{jensen1969much} These two tendencies—to assign immigrant children to special education schools and not to assign them—would cancel each other out. It is currently unknown how large these biases are for the Danish data, so some caution is advised.

2. Your quote would have more "punch" if it came from a mainstream newspaper. But maybe that's too much to hope for.

I am willing to approve the manuscript. Do you have enough approvals for publication?
Admin
Emil,

1. I rewrote the paragraph to emphasize the point that the two tendencies would cancel each other out. Feel free to accept or reject what I wrote:

This study only concerns normal schools and excludes special education schools, which have a higher proportion of lower GI students. Immigrants may thus be over-represented in such schools, thereby reducing the GPA gap in normal schools and making it seem smaller than it really is for the Danish population as a whole. Immigrants are in fact over-represented in some special education schools, especially the second generation (see the appendix for details). On the other hand, there may be under-diagnosis of special education needs among immigrant students. Such under-diagnosis would happen partly because social workers may try to avoid assigning immigrant students to schools for the retarded and partly because a higher proportion of immigrants appear behaviorally normal even in the "retarded" range (usually less than an IQ of 75). In a population with a lower mean IQ, it is less likely that a below-75 IQ would have unusual genetic or developmental causes (e.g., Down's Syndrome) that disrupt many aspects of mental functioning. Low IQ individuals would thus seem more behaviorally normal and be less likely assigned to special education classes.\cite{jensen1969much} These two tendencies—to assign immigrant children to special education schools and not to assign them—would cancel each other out. It is currently unknown how large these biases are for the Danish data, so some caution is advised.


Thanks. Your version is better. I have replaced mine with yours.

2. Your quote would have more "punch" if it came from a mainstream newspaper. But maybe that's too much to hope for.


Most journalists are very left-leaning in Denmark. I have reviewed the surveys of this before and found that 70-90% of journalists or journalist students voted or would vote for left-wing parties excluding the conservative Danish People's Party which altho it has a leftist economic policy, has a conservative social policy.

However, if one reads the comment sections on newspaper, one will see that people frequently invoke Islam or some connected feature with it such as parallel societies, counter-culture and the like. E.g. counter-culture is mentioned here.

Mostly, the discussion of stuff like this takes place in the comment sections and on user-driven sites such as 180grader.dk. This is

Sometimes one will see people denying the influence of Islam on negative outcomes. E.g. one Muslim debater denies the influence of Islam on crime rates among Muslim groups here.

Even if not mentioned in mainstream media, it deserves attention.

I don't want to expand the paper to discuss possible media bias or the leftism of journalists etc., I want to stay closer to the numbers and avoid the more political aspects.

I am willing to approve the manuscript. Do you have enough approvals for publication?


No, I have two approvals: one for Fuerst and one from you. Meng Hu approved an earlier version, however this paper has changed so much that a re-approval seems in order. I will send him an email.

I have updated the draft at OSF.
I read the new version (v8) and I don't disagree with anything in the entire text. I re-approve. But even if I disagreed with any of the modified portions, due to reviews, I don't think I will disapprove because it means I have to discuss the matter with the reviewer(s) in question. I don't think it's reasonable to go so far (and it's very complicated for obvious reasons I don't need to tell). For such modifications (and only for this kind), then, the authors don't need my re-approval.

In fact, I only have some quibbles :

The use of GI (General Intelligence) while most authors would have written GCA (General Cognitive Abilities). They mean just the same thing. Why not using GCA ? Because using another, new term is not very practical (and in fact it's very irritating in my opinion) to use different terms for saying just the same thing. Say GCA and everyone in the field understands what you say. But say GI, and no one could answer that without asking the meaning of it.

Since GI is not the only factor that causes differences in GI, one would not expect a GI difference of 1.0 d to be associated with a 1.0 d difference in GPA.


1) The second GI should be GPA. 2) It can even if more than one variable affects GPA, since a variable that would have decreased the correlation IQ-GPA could have been masked (offset) by another variable that would have increased the correlation IQ-GPA.

And in page 6, you wrote "The GPA gaps were -1.7 and -1.3, -.50 and -.38 d respectively" while in your table 4 the gaps are 1.7, 1.3, -0.50 and -0.38.

The value for the second generation is smaller because this group does better in school while the GI gap is very similar, even a bit larger.


It's overoptimistic. I don't call a difference of 0.8 "even a bit larger". There's no change. That's all.

Tables 2 & 5 : same problem as before. Predictor should be used for indepedent vars inserted into the regression equation in a regression analysis, as its definition implies. There is no "predictor" in a correlation analysis because both variables are treated the same. And at page 7, you wrote "predictor analyses". I know what is a regression analysis and a correlation analysis but I doubt anyone knows what is a predictor analysis.

In table 2 by the way, some numbers have 2 digits after comma, some have 1 digit, and another has zero digit. I think it's preferable to have all numbers with the same number of digit. Even for the zero correlation.

or that the particular group has increased/decreased its GI in Denmark due to improved environment.


The way you write it is confusing because it implies that improved environment can either increase or decrease IQ. If a good environment has an effect on IQ, it's through IQ gain.

I examined this for all countries in both datasets and found the median value (to avoid effects of outliers).


But outliers are sometimes useful. The more you have outliers, and the more likely you would have missed an important information, i.e., a confounding variable. And even a single outlier can be of some interest, sometimes, if you know/understand the reason for this behavior.

thereby reducing the GPA gap in normal schools and making it seem smaller than it really is for the


seem with an "s".
Admin
Thanks for taking the time to review again.

I read the new version (v8) and I don't disagree with anything in the entire text. I re-approve. But even if I disagreed with any of the modified portions, due to reviews, I don't think I will disapprove because it means I have to discuss the matter with the reviewer(s) in question. I don't think it's reasonable to go so far (and it's very complicated for obvious reasons I don't need to tell). For such modifications (and only for this kind), then, the authors don't need my re-approval.

In fact, I only have some quibbles :

The use of GI (General Intelligence) while most authors would have written GCA (General Cognitive Abilities). They mean just the same thing. Why not using GCA ? Because using another, new term is not very practical (and in fact it's very irritating in my opinion) to use different terms for saying just the same thing. Say GCA and everyone in the field understands what you say. But say GI, and no one could answer that without asking the meaning of it.


You are right. I have changed it to GCA. I have also changed the "general intelligence" to "general cognitive ability", and added some more keywords.

1) The second GI should be GPA. 2) It can even if more than one variable affects GPA, since a variable that would have decreased the correlation IQ-GPA could have been masked (offset) by another variable that would have increased the correlation IQ-GPA.

And in page 6, you wrote "The GPA gaps were -1.7 and -1.3, -.50 and -.38 d respectively" while in your table 4 the gaps are 1.7, 1.3, -0.50 and -0.38.


I have fixed the GI to GPA. I have added the minus signs to the table.

It's overoptimistic. I don't call a difference of 0.8 "even a bit larger". There's no change. That's all.

Tables 2 & 5 : same problem as before. Predictor should be used for indepedent vars inserted into the regression equation in a regression analysis, as its definition implies. There is no "predictor" in a correlation analysis because both variables are treated the same. And at page 7, you wrote "predictor analyses". I know what is a regression analysis and a correlation analysis but I doubt anyone knows what is a predictor analysis.

In table 2 by the way, some numbers have 2 digits after comma, some have 1 digit, and another has zero digit. I think it's preferable to have all numbers with the same number of digit. Even for the zero correlation.


Calling something an independent/predictor has to do with what role it is used in. Correlations are the same as regressions when the variables are standardized. I will keep the wording.

With "predictor analysis" I meant the ones where I tried to predict the GPA differences, as opposed to the, i.e. those in section two (title = "Predictive analyses"). I should of course have written that, so I have changed it now.

The way you write it is confusing because it implies that improved environment can either increase or decrease IQ. If a good environment has an effect on IQ, it's through IQ gain.


You are right. I have removed the "decrease".

But outliers are sometimes useful. The more you have outliers, and the more likely you would have missed an important information, i.e., a confounding variable. And even a single outlier can be of some interest, sometimes, if you know/understand the reason for this behavior.


Sure, but with small datasets such as these, I'm more interested in the general tendencies. Outlier analysis is dangerous when datasets are small.

seem with an "s".


Not sure what you mean. The "seem" is correct without an 's' here.

---

I have uploaded a new draft. I will give MH a chance to respond to this altho I have 3 approvals now.

I made some changes to formulations here and there and added two more references (concerning language bias in testing).

Version 9: https://osf.io/p9d5z/
I will give MH a chance to respond to this altho I have 3 approvals now.


There is no need for this. If I approved, it's because whether or not these objections were addressed, that won't change my mind. And I don't want to argue indefinitely about these points.
1) The second GI should be GPA. 2) It can even if more than one variable affects GPA, since a variable that would have decreased the correlation IQ-GPA could have been masked (offset) by another variable that would have increased the correlation IQ-GPA.

And in page 6, you wrote "The GPA gaps were -1.7 and -1.3, -.50 and -.38 d respectively" while in your table 4 the gaps are 1.7, 1.3, -0.50 and -0.38.


That strikes me as being a non-trivial error. I didn't catch it because I usually don't check the tables. Could the author double check all of his numbers? Thanks.
Admin
The error is trivial. It just has to do with whether or not I reverse coded the numbers.
The error is trivial. It just has to do with whether or not I reverse coded the numbers.


If you are satisfied with the paper, then publish it at your convenience.
Admin
I have uploaded a publication version (no longer marked as a draft). https://osf.io/p9d5z/

I will publish it later today.