Imagine we had 5 NAEP Reading tests. And 2 of them showed no DIF and no measure non-invariance. Imagine that the H/W difference on these 2 tests was 0.65. This would provide evidence that there was a "true" population level latent H/W ability difference of 0.65 SD, no? Now imagine we had 3 other NAEP tests for which DIF and measure non-invariance were found, but adjusted scores were not presented. Imagine that these 3 tests also showed an average score difference of 0.65 SD. Knowing nothing else, we can infer that the psychometric bias on the latter 3 tests is not accounting for much of the average 0.65 H/W score difference because the evidence show that there is, in fact, a 0.65 SD latent ability difference.

I understand what you mean, And i think I said something like this earlier. My point is that your argument is correct only if the first 2 NAEP (invariant; d=0.65) and last 3 NAEP (non-invariant; d=0.65) test have the same or similar properties (note; if that happens, there is some indirect evidence that the bias is not cumulative, while I'm talking about cumulative bias, since non-cumulative bias is generally irrelevant when it concerns IQ). Why I said earlier that most people (including practioners) do not understand what measurement equivalence/invariance is, has to do with test composition. When MI is violated, this means that the group difference differs depending on the kind of subtest/items the test is composed of. This is probably why Wicherts said something like "scores are not comparable" when MI is not fulfilled. It's not 100% wrong, but it's highly misleading.

However, when you are talking about tests tapping into different cognitive dimensions (e.g., reading vs. math, achievement vs. IQ) the assumption that the tests have similar properties is very likely to be violated. And in this case, generalizability becomes impossible.

-----

Oh, concerning the median/mean issue...

You can use the following

=MEDIAN(F47:F64)

When you do this, the d gap for blacks are 1.05, 0.85, 1.00, while using means you have 0.98, 0.80, 0.98.

Hmm, let me see. For hispanic, means are 1.02, 0.68, 0.56, and for medians, 0.90, 0.68, 0.56. For asian, means are 0.10, -0.21, -0.19, and for medians, 0.05, -0.20, -0.16. In other words, it seems to make an effect only for 1rst generations. The numbers for means differ from the last version of the paper, because (I assume) the version of the data set I have is outdated and modifications should have been made so far.

Below is, again, based on the spreadsheet data I have.

For blacks, the values are wide, between 0.50 and 1.60. I'm not sure there is a particular outlier here.
For hispanics, it's also difficult to tell because the HSLS2009 has 0.37 while other values range between 0.60 and 1.80-1.90.
For asians, outliers should be the Add Health (0.82) and GSS (0.85). Interestingly, those surveys have small samples for asians.

[quote][/quote]

I made the corrections noted by Emil; I checked over the tables and made corrections where needed. I added medians and noted: "On a reviewer’s request, median d-values were also reported in Table 2."

Regarding MH's point, there is nothing I can say more. Richwine's analysis of the CNLSY PIAT provides evidence that correcting for bias does not substantially reduce the (math) d-values. MH thinks that this is irrelevant because the tests used in my meta-analysis are "too different". But PIAT MATH is a good measure of math ability and most of my tests (14/18) involved Math tests.

BPSS-- SAT/ACT (Math + Reading)
NPSAS2012* -- SAT/ACT (Math + Reading)
NPSAS2008* -- SAT/ACT (Math + Reading)
TIMSS Grade 4 2007 -- (Math + Science)
TIMSS Grade 4 2011-- (Math + Science)
TIMSS Grade 8 2007 -- (Math + Science)
TIMSS Grade 8 2011 -- (Math + Science)
PISA 2009 -- (Reading + Math + Science)
PISA 2012 -- (Reading + Math + Science)
NAAL 2003 -- (Numeracy + Literacy)
PIAAC 2012 -- (Numeracy + Literacy)
NLSF -- SAT/ACT (Math + Reading)
HSLS 2009 -- (Math test)
NELS88 -- (Math test)

That said, I am willing to allow MH to rewrite the passage in a way that works for him.

...

Now, this apart, there is only one other issue. Most of my tests are "crystal" or "informational". If this is a problem, I could note PISA creative problem solving results and quickly compare these to the PISA R/M/S results. I would have to have MH compute the d-values, though. I don't recall there being a large difference, though.

I approve the publication, too, but I have a few more quibbles that you may want to address:

1) "Submitted to Open Differential Psychology June 29nd, 2014"

29th

2) A better title would be "Ethnic/Race Differences in Aptitude by Generation in the United States: An Exploratory Meta-analysis".

3) "subpopulations within these broad categories need not perform as do the racial/ethnic groups do on average"

one "do" too many

3) For better readability, you should justify paragraphs so that both sides are aligned, and use a bigger font size for main headings as compared to sub-headings.

4) "They found that, relatives to stayers"

relative to

5) "Negatively selection, then, is not likely"

Negative selection

6) Regarding the IQs of different Asian American groups, you might want to cite Hsin & Xie (2014) (attached) who found that Filipinos and South Asians had lower math and reading test scores than whites nationally (see Fig. 4).

6) Regarding the IQs of different Asian American groups, you might want to cite Hsin & Xie (2014) (attached) who found that Filipinos and South Asians had lower math and reading test scores than whites nationally (see Fig. 4).

Thanks for the feedback, D. I spent the day adding 4 more TIMSS results to increase reliability (combining new and old). The results were effectively the same. Just a lot of extra work. As for Hsin and Xie, they present within school differences i.e., controlling for school fixed effects. OK. But I imagine that this also controls for cognitive ability. You end up effectively matching on a bunch of unobserved factors. Also, the ECLS results seem to be all over the place.

I'm quite fond of my terse reviews. Lynn reviews in the same way. If you'd like me to review more thoroughly, however, I will do so.

(updated 5:10, 7/19/2014) Document, PDF, Excel, Appendix,

(Per D's request, I added Asian subgroup scores based on three National samples (see: pages 26-28). The results were substantively the same as the California CAT ones. N.E. Asians > S.E. Asians by about 0.5 SD; Lynn's NIQs substantially under-predict S.E. and S. Asian American scores.)

I am waiting for MH's approval.

(Oh, and thanks for the help with this; I'll try not to be such a mess next time.)

(Busy those days, but this thread seems to be very active during this period)

I will read the newest version. But I will, if you don't mind wait just a little bit. I have a question I want to ask to Lynn concerning asian scores (pp 21-23). I will give you my final verdict after this (but I can tell you the last version I have read (2 days by now) was excellent and was probably 99% ready for publication.

With regard to the measurement bias, when i said "too different" (or so) I did not say the assumption is necessarily wrong. But I need to compare the kind of items in the TIMSS (and others) to that used in the PIAT math, for example, and if the question are similar, then I agree that Richwine conclusion can be extended outside the PIAT math. By way of comparison, I think the following comment here "Third, the differences between English-only speaking 2nd+ generation Hispanics and Whites is about as large as the difference between all 2nd+ generation Hispanics and Whites." is much more convincing than you invoking Richwine or Trundt et al. studies.

EDIT : Oh, before i forget. There is a passage that needs be rewritten :

The data used was originally presenting in Pang et al. (2011).

I think it should be presented.

Lynn has not responded. I don't want to wait more, so I will let you know my comments.

For TIMSS grade 4 2003. I see you have not used it in your averages. Perhaps you can still use the following formulas i have made in order to show that it will not change your estimates based on 1999-1995 averages.

TIMSS MATH grade 4 2003

=(E$168-H168)/G$168
=(E$168-H169)/G$168
=(E$168-E169)/G$168
=(E$168-H171)/G$168
=(E$168-E171)/G$168
=(E$168-K168)/G$168
=(E$168-K169)/G$168
=(E$168-K171)/G$168

=(E$168-N169)/G$168
=(E$168-N171)/G$168


TIMSS SCIENCE grade 4 2003

=(E$188-H188)/G$188
=(E$188-H189)/G$188
=(E$188-E189)/G$188
=(E$188-H191)/G$188
=(E$188-E191)/G$188
=(E$188-K188)/G$188
=(E$188-K189)/G$188
=(E$188-K191)/G$188

=(E$188-N189)/G$188
=(E$188-N191)/G$188

(You see there is an empty line, it's because of missing data)

I see the second row in table 1 should be SAT/ACT.

Concerning TIMSS 2003 math (grade 4) there is a spreadsheet formula that is wrong: look at cells P219 and P221. The correct formulas, respectively, should have been:

=(E$168-K168)/G$168
=(E$168-K171)/G$168

The correct values (see column R) for hispanic 3rd and 1st are .60 and 1.50 for 2003. And the correct values for the average 2003-1995 become .64 and 1.27, instead of .69 and 1.22. I sent you the corrected spreadsheet by mail. In your spreadsheet "meta-result" you see the average d value did not change despite the changes made above. So, you don't need to change these numbers in your article. I have, by the way, added the median d values in your spreadsheet.

Page 25, you type, "This was significant at r (10) = 0.75, p < 0.05 and rho (10) = 0.59, p < 0.05." What's in parenthese ? Df ? Perhaps you should rewrite it as r (df=10) = 0.75 ?

Table 16, for filipinos, column G, in your paper, you have 0.80 but in your spreadsheet, it's 0.79. But I don't think it's really a problem.

Now, my last request concerns table 9. I haven't looked at it before, but now, when I examine the d values for your spreadsheet analysis 6 (PIAAC 2012), i don't see those numbers. It seems I missed something. If that comes from another published paper, as it seems, i would like you add the reference.

For TIMSS grade 4 2003. I see you have not used it in your averages. Perhaps you can still use the following formulas i have made in order to show that it will not change your estimates based on 1999-1995 averages.

I combined the grade 4 TIMSS 2003 with the grade 4 TMSS 1995.

I see the second row in table 1 should be SAT/ACT.

Already fixed this.

Concerning TIMSS 2003 math (grade 4) there is a spreadsheet formula that is wrong: look at cells P219 and P221. The correct formulas, respectively, should have been:

Thanks. You didn't send the correct .exl file but I found the error and changed it. I changed by Document File.

Page 25, you type, "This was significant at r (10) = 0.75, p < 0.05 and rho (10) = 0.59, p < 0.05." What's in parenthese ? Df ? Perhaps you should rewrite it as r (df=10) = 0.75 ?

Added df.

Table 16, for filipinos, column G, in your paper, you have 0.80 but in your spreadsheet, it's 0.79. But I don't think it's really a problem

I changed the excel to 0.80.

Now, my last request concerns table 9. I haven't looked at it before, but now, when I examine the d values for your spreadsheet analysis 6 (PIAAC 2012), i don't see those numbers. It seems I missed something. If that comes from another published paper, as it seems, i would like you add the reference.

The d-values for PIACC are listed in column D of table 9; they match with the d-values in table 2. The other values (English ability) are from a separate analysis. I discussed that analysis in the foregoing paragraph. This was a micro-sub point, so I didn't create special tables in the excel file.

Are we all good? If so, I will upload the latest copy.
[hr]

Sample size I think. r=.75 with N=10 gives (two-tailed) p=0.01247787.

N=12, Df = N-2 = 10.

I'm Ok with the changes. Is it possible to publish the last version ? (Don't forget to correct for the error in table 16. You have typed "Laoations" instead of "Laotians".)

I'm Ok with the changes. Is it possible to publish the last version ? (Don't forget to correct for the error in table 16. You have typed "Laoations" instead of "Laotians".)

Attached.

It is still not "SAT/ACT" in row 2 of table 1.

And, one question before I finish with this. In your spreadsheet (try to correct sheet "analyses" where the rows/columns are not visible, if possible), I still don't see the analysis that gives you the columns A, B and C of your table 9 (PIAAC 2012).

Concerning TIMSS grade 4 2003, I should have said grade 8. (By the way, the formulas I have given above were for grade 4, not 8. Don't understand why I made this mistake.)

(a) "It is still not "SAT/ACT" in row 2 of table 1."

I don't know what you're talking about. Take a screenshot and circle the error.

(b) "And, one question before I finish with this. In your spreadsheet (try to correct sheet "analyses" where the rows/columns are not visible, if possible), I still don't see the analysis that gives you the columns A, B and C of your table 9 (PIAAC 2012)."

I didn't include it. And I am not going to.

(c) "Concerning TIMSS grade 4 2003, I should have said grade 8. (By the way, the formulas I have given above were for grade 4, not 8. Don't understand why I made this mistake.)"

I checked it and there was a TIMSS grade 4 2003 Hispanic 1st and 3rd generation error. The numbers you said were correct. And I fixed it. Are you saying that there was also a grade 8 error? I don't understand because I didn't include grade 8 2003 data in the meta.

(d)

"try to correct sheet "analyses" where the rows/columns are not visible, if possible"

The reader just has to click on "headings--> view" I am sure that they can figure it out.

(a) "It is still not "SAT/ACT" in row 2 of table 1."

I don't know what you're talking about. Take a screenshot and circle the error.

(b) "And, one question before I finish with this. In your spreadsheet (try to correct sheet "analyses" where the rows/columns are not visible, if possible), I still don't see the analysis that gives you the columns A, B and C of your table 9 (PIAAC 2012)."

I didn't include it. And I am not going to.

(c) "Concerning TIMSS grade 4 2003, I should have said grade 8. (By the way, the formulas I have given above were for grade 4, not 8. Don't understand why I made this mistake.)"

I checked it and there was a TIMSS grade 4 2003 Hispanic 1st and 3rd generation error. The numbers you said were correct. And I fixed it. Are you saying that there was also a grade 8 error? I don't understand because I didn't include grade 8 2003 data in the meta.

(a) See here.
http://www.openpsych.net/forum/attachment.php?aid=297
Page 5, table 1, the row that belongs to NPSAS2012, under column "test" you have "SATACT".

(b) that's a problem because I'm actually trying to replicate those numbers but I can't. For example when I factor analyze (PAF) all the variables you use in your syntax, the gap for hispanic 1st gen vs white 3rd, is 0.45. It's not the tremendous gap of -3.86. A factor score is expressed in z score no ? 4 is just too much for me to believe it.

(c) i said I did not understand why you use 2003-1995 for grade 4 but 1999-1995 for grade 8, without reporting the d gaps for the year 2003. It's just to see if you will get different values using 2003 instead of other years.