Back to [Archive] Post-review discussions

[ODP] The Scandinavian WAIS IV Matrices as a Test of Dutton, te Nijenhuis and Rovaine
I have now edited this to reply to Meng Hu's concern over the differences between WAIS IV matrices and PISA CPS. According to Rindermann's (2007) paper there is a correlation of 0.97 between PISA Problem Solving 03 and the matrices part of IQ tests. In addition, Pearson's and the OECD's descriptions of these tests make clear that they are extremely similar.


Where is PISA CPS 2003? As far as I know, PISA CPS was first administered in 2012. At least I am not aware of any published scores prior to 2012.
Admin
Rindermann 2007 is given as:

Rindermann, H. (2007). The g-Factor of International Cognitive Ability Comparisons: The Homogeneity of Results in PISA, TIMSS, PIRLS and IQ-Tests Across Nations. European Journal of Personality, 21: 667-706.

Table 1 does have a PISA Problems-03.

There is apparently a PISA03 PS (no word about creative or not).

http://www.oecd.org/dataoecd/25/12/34009000.pdf

Data is on table 2.2, PDF page 140.

I added them to the Megadataset. Some correlations:

Variable PS03Mean PS03SD CPS12Mean CPS12SD LV12IQ
ProblemSolving03Mean 1.00 0.06 0.86 0.18 0.96
ProblemSolving03SD 0.06 1.00 -0.38 0.15 0.10
CPS12Mean 0.86 -0.38 1.00 -0.20 0.90
CPS12SD 0.18 0.15 -0.20 1.00 0.01
LV2012estimatedIQ 0.96 0.10 0.90 0.01 1.00


So the unadjusted correlation is .96, not .97.

Other correlations of interest: CPS12 x PS03 = .86, but CPS12SD x PS03SD = .18. PISA does not measure SD reliably.
Uh oh. Better not tell the media that PISA tests are IQ tests!


Thing is, I've already done that. In fact, they'll be an article about in the Times (Educational Supplement) on, I think, Friday.
Rindermann 2007 is given as:

Rindermann, H. (2007). The g-Factor of International Cognitive Ability Comparisons: The Homogeneity of Results in PISA, TIMSS, PIRLS and IQ-Tests Across Nations. European Journal of Personality, 21: 667-706.

Table 1 does have a PISA Problems-03.

There is apparently a PISA03 PS (no word about creative or not).

http://www.oecd.org/dataoecd/25/12/34009000.pdf

Data is on table 2.2, PDF page 140.

I added them to the Megadataset. Some correlations:

Variable PS03Mean PS03SD CPS12Mean CPS12SD LV12IQ
ProblemSolving03Mean 1.00 0.06 0.86 0.18 0.96
ProblemSolving03SD 0.06 1.00 -0.38 0.15 0.10
CPS12Mean 0.86 -0.38 1.00 -0.20 0.90
CPS12SD 0.18 0.15 -0.20 1.00 0.01
LV2012estimatedIQ 0.96 0.10 0.90 0.01 1.00


So the unadjusted correlation is .96, not .97.

Other correlations of interest: CPS12 x PS03 = .86, but CPS12SD x PS03SD = .18. PISA does not measure SD reliably.


The items are very similar to those of CPS. I think it's essentially the same stuff, just a fancy name for the new version.
Edit 1: I do not know if the items with the lowest reliability were removed from PISA CPS 2012 though...the 2003 stuff seems pretty preliminary.
Admin
I forgot to use the measured IQ variable instead of the combined scholastic+measured IQ variable.

Here's the same results with both IQ variables.

Variable CPS12Mean PS03Mean LV2012measuredIQ LV2012estimatedIQ
CPS12Mean 1.00 0.86 0.89 0.90
PS03Mean 0.86 1.00 0.97 0.96
LV2012measuredIQ 0.89 0.97 1.00 1.00
LV2012estimatedIQ 0.90 0.96 1.00 1.00


The PS03 x measured IQ is .97, as stated by Rindermann. Interestingly the CPS03 correlates more strongly with measured IQ than does CPS12 (.97 vs. .89). Due to more inclusive sample perhaps?

Note however, that Rindermann used the LV2006 IQ data, and I used the LV2012 one here with modifications from Jason Malloy's results.

Note also that the correlation between the 2 IQ measurements is not exactly 1.00, these are rounded to 2 decimals. The cor is 0.9995575.
1) You should mention the PISA results, too, in the abstract.

2) "It can be seen that in all but one age group the average Scandinavian score on the WAIS IV matrices is higher than the Finnish."

By my count, Finnish scores (US norm) are higher in three age groups, Scandinavian scores in six age groups, one age group has a missing value for Finns, and there's one tie. The Scandinavian advantage exists for older cohorts, there's little difference among those below 40.

3) "There is a body of evidence that this minority are genetically closer to the Swedes than to the Finns (e.g. Virtaranta-Knowles et al., 1991)"

That's a peculiarly old genetics reference. Here's a newer one, which says:

Among our Finnish sample, genetically closest to Swedes were the Swedish-speaking Finns of coastal Ostrobothnia. This agrees well with the history of the Swedish-speakers, who arrived into the western and southern coastal areas of Finland in the beginning of the second millennium [21]. However, they have obviously experienced a lot of subsequent admixture with the Finnish-speakers, resulting in a subtle difference between them and their closest neighbors; conversely, their genetic distance from the Swedes is of the same magnitude as the largest distances between provinces within Sweden. A similar, intermediate position of the Swedish-speakers has been detected earlier [22], although with differing admixture proportions, probably depending on the choice of reference samples.


It's probably more accurate to say the Swedish-speaking minority is intermediate between Finns and Swedes. The old reference puts Finnish admixture in the Swedish-speakers at 60 percent.

4) The source for your PISA data is not specified.
I said that I don't understand why the WAIS matrices subtest is dismissed just because it has smaller N. Generally, what people do is to average the two samples. This point is still not discussed in the article.

That the two tests have high correlation is a good thing (although you'll need to add this information in the article). It indicates they measure the same thing. This is only part of the answer. Two tests can be correlated at 100% but this tells us nothing about how the group would have scored. It tells us only that the rank order is not altered. See here. I was not merely talking about correlations. I was also thinking there might be some particularity in the test that makes Finland scoring lower in WAIS matrices than scandinavian countries, or scoring higher in PISA CPS than scandinavian countries. It's just speculation about the possible causes of the differences. But if really the two tests are psychometrically equivalent, this gives even less reason to dismiss the lower score among finnish people on the WAIS matrices.

EDIT : I forgot to tell you that you should correct this little error :

However, as the education systems are relatively similar (see Kananen, 2014) it is unclear how this might be the case[b]..[/b]
Admin
I fixed the double dot. I will let Dutton decide what to do with the averaging question. MH is right that averaging is the correct solution. Averaging is a kind of meta-analysis which is always preferable to single studies no matter how comprehensive.

That r=1.0 tells us more than the rank order. It also tells us that the relative size of the intervals is the same. This is not the case for rank-order correlations (Spearman).
I don't know what's going on with the F-S and am working on a paper on them, which I can mail you a first draft of if it interests you.


Can you attach it here? I am interested too...or email it to me please.


I will send you over a copy of the FS paper when this one is finished
We have now conducted the meta-analysis suggested by Meng Hu and made all the changes suggested by Dalliard. I also noted that there is a case for arguing that PISA is more representative than WAIS IV because its exclusion criteria is less strict.
I've realized that the paper doesn't justify why Finland is compared to Scandinavia and not say, Germany or Spain? Is it because they are geographically close? Genetically speaking, Scandinavian countries are more similar to Northwestern Europe than to Finland. I think the paper should specify why the author chose to compare Finland against Scandinavia or viceversa, why Scandinavia was compared to Finland and not to another European country.
Another thing: as tentative as my genetic method can be, the Finns have the highest score in Europe on the principal component for educational attainment alleles. See table 2 of my paper: http://figshare.com/articles/Detecting_polygenes_using_signals_of_polygenic_selection_Tools_for_increasing_the_power_of_GWAS_/1172301

Perhaps cite this?
It'd be good to have genes for fluid g to compare this with but for now we don't.

Piffer, Davide; Gilfoyle, Bertram (2014): Detecting “polygenes” using signals of polygenic selection. Tools for increasing the power of GWAS.. figshare.
http://dx.doi.org/10.6084/m9.figshare.1172301
Ok, i've made those changes.

Another thing: as tentative as my genetic method can be, the Finns have the highest score in Europe on the principal component for educational attainment alleles. See table 2 of my paper: http://figshare.com/articles/Detecting_polygenes_using_signals_of_polygenic_selection_Tools_for_increasing_the_power_of_GWAS_/1172301

Perhaps cite this?
It'd be good to have genes for fluid g to compare this with but for now we don't.

Piffer, Davide; Gilfoyle, Bertram (2014): Detecting “polygenes” using signals of polygenic selection. Tools for increasing the power of GWAS.. figshare.
http://dx.doi.org/10.6084/m9.figshare.1172301
Ok, i've made those changes.

Another thing: as tentative as my genetic method can be, the Finns have the highest score in Europe on the principal component for educational attainment alleles. See table 2 of my paper: http://figshare.com/articles/Detecting_polygenes_using_signals_of_polygenic_selection_Tools_for_increasing_the_power_of_GWAS_/1172301

Perhaps cite this?
It'd be good to have genes for fluid g to compare this with but for now we don't.

Piffer, Davide; Gilfoyle, Bertram (2014): Detecting “polygenes” using signals of polygenic selection. Tools for increasing the power of GWAS.. figshare.
http://dx.doi.org/10.6084/m9.figshare.1172301


Excellent. I confirm my ok to publication.
The new version is good. It contains the changes I requested.

I approve.

However, I hope you will do another analysis when you have more data. Because I'm still half-convinced that finnish people should have higher Gf. The only reason why you find this result is because you have weighted the data set. But the N for WAIS matrices subtest is not small. Generally, I think sampling error is a threat when N is small or very small. That's not the case in your data, so I don't think sampling error is the explanation for the result. The only way to answer this question is to do a follow-up on additional data, and ideally, based on still other test/subtests of Gf type.

Furthermore, I would like you to correct this error :

To arrive at a single best estimate, we performed a meta-analysis (N-weighted) based both data sets.


The color text for the reference Salmela 2011 is in grey while it should be in black.

And finally, I want to say I don't necessarily like the use of google docs for your drafts. When you modify the draft, the old version(s) is(are) not visible anymore due to the modification/deletion. I prefer to upload all my versions (1, 2, 3, 4, etc.) in OSF, so that everyone can see the progression of my work. I know this journal says nothing about it, but that's my suggestion.
Admin
I fixed the missing word "on" and the grey color errors.

And finally, I want to say I don't necessarily like the use of google docs for your drafts. When you modify the draft, the old version(s) is(are) not visible anymore due to the modification/deletion. I prefer to upload all my versions (1, 2, 3, 4, etc.) in OSF, so that everyone can see the progression of my work. I know this journal says nothing about it, but that's my suggestion.


Google Drive has versioning (so one can follow the revisions). The trouble is that it is not publicly available. I can see it when I'm logged in, but not when logged out. https://support.google.com/docs/answer/190843?hl=en

Thus I agree that it is a problem, it is also a very useful program.

The entire purpose of the review threads is for people to be able to follow the review process. They cannot do it properly when the old versions are hidden away like this.

On the other have, GDrive is very useful when collaborating on a project where multiple people take part in the writing and fixing of errors as is the case here. This would be much more difficult if one had to rely on uploading new drafts to OSF all the time. GDrive is a more fluid experience.

So what to do? One idea is to make copies of the GDrive document before making major changes to it, so as to have one for each version. This is not optimal in that it will clog up the results when searching in GDrive with copies.

Another idea is to periodically export copies from Drive and save them on OSF. Preferably before any major change. This is perhaps the most reasonable method.
Another idea is to periodically export copies from Drive and save them on GDrive. Preferably before any major change. This is perhaps the most reasonable method.


I think before any major change, copies should be saved and uploaded to the forum together with the updated version.
Admin
Another idea is to periodically export copies from Drive and save them on GDrive. Preferably before any major change. This is perhaps the most reasonable method.


I think before any major change, copies should be saved and uploaded to the forum together with the updated version.


I misstated my idea in the quoted text. I meant uploaded to OSF, not Drive. OSF is better for versioning than forums.
I approve this version.
Another idea is to periodically export copies from Drive and save them on OSF. Preferably before any major change. This is perhaps the most reasonable method.


That's a good idea.