[ODP] - Parents’ Income is a Poor Predictor of SAT Score
2014-Jun-04, 00:49:06, (This post was last modified: 2014-Jun-11, 06:46:42 by nooffensebut.)
#1
[ODP] - Parents’ Income is a Poor Predictor of SAT Score
Author: nooffensebut

Abstract

Parents’ annual income lacks statistical significance as a predictor of state SAT scores when additional variables are well controlled. Spearman rank correlation coefficients reveal parents’ income to be a weaker predictor of average SAT scores for each income bracket within each state than parents’ education level as a predictor of average SAT scores for each education level within each state. Multiple linear regression of state SAT scores with covariates for sample size, state participation, year, and each possible combination of ordinal variables for parents’ income, parents’ education, and race shows income to lack statistical significance in 49% of the iterations with greater frequency of insignificance among iterations with higher explained variance. Cohen’s d comparisons of the yearly individual SAT advantage of having educated parents shows a fairly consistently increasing positive relationship over time, whereas similar analysis of the yearly individual SAT advantage of having high-income parents shows variability somewhat coinciding with the business cycle.

Key words: SAT; socioeconomic status; income; education; race

(Update: I made slight stylistic/aesthetic changes to pdf - 6/6/2014)
(Update: Nixed Hampshire et al. Added MacCallum et al. - 6/10/2014)


Attached Files
.pdf   Parents Income is a Poor Predictor of SAT Scores - ODP.pdf (Size: 1.29 MB / Downloads: 1,884)
.xlsx   supplement 1 - education data.xlsx (Size: 242.46 KB / Downloads: 226)
.xlsx   supplement 2 - income data.xlsx (Size: 303.62 KB / Downloads: 243)
.xlsx   supplement 3 - states continuous variables data.xlsx (Size: 146.54 KB / Downloads: 221)
Reply
2014-Jun-04, 10:05:19,
#2
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
(2014-Jun-04, 00:49:06)nooffensebut Wrote: Author: nooffensebut

Abstract

Parents’ annual income lacks statistical significance as a predictor of state SAT scores when additional variables are well controlled. Spearman rank correlation coefficients reveal parents’ income to be a weaker predictor of average SAT scores for each income bracket within each state than parents’ education level as a predictor of average SAT scores for each education level within each state. Multiple linear regression of state SAT scores with covariates for sample size, state participation, year, and each possible combination of ordinal variables for parents’ income, parents’ education, and race shows income to lack statistical significance in 49% of the iterations with greater frequency of insignificance among iterations with higher explained variance. Cohen’s d comparisons of the yearly individual SAT advantage of having educated parents shows a fairly consistently increasing positive relationship over time, whereas similar analysis of the yearly individual SAT advantage of having high-income parents shows variability somewhat coinciding with the business cycle.

Key words: SAT; socioeconomic status; income; education; race

This is a very good paper and provides original insights. Thus, I recommend publication.
Reply
2014-Jun-08, 01:56:17, (This post was last modified: 2014-Jun-08, 02:01:33 by Meng Hu.)
#3
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
I come a little bit late, because when I saw this submission I was thinking what to do about my article on multiple regressions, since lot of people don't understand what it is.

http://humanvarieties.org/2014/06/07/mul...fallacies/

As you see the title is provocative that's why I needed a little more time. I think everyone should read it. It's unlikely (~0.00000000001) that I'm wrong.

I believe it might me useful since you talk about regressions. Thus this needs to be well interpreted. For example the following statement

Quote:β standardized coefficients for each covariate show that parents’ education predicts state SAT scores best, followed by state SAT participation rate.

is incorrect, at least in the sense you seem to interpret the "predictors" which are "predictors" in name only, but not in reality.

What regression does it to estimate the direct effect of your independent variable, not the total effect, which is composed of direct and indirect effects. In fact multiple regression cannot disentangle the pattern of indirect effects. The only way to disentangle these indirect effects is to use longitudinal data with repeated measures, with tools such as path analysis or SEM.

This is a fact that most scientists ignore. Indeed, you can see that my blog post is somewhat like a declaration of war. That's why I hesitated to post it.

Now that is said your article has really good quality. i haven't finished yet, I'll come back later.

(Also, if you can write a blog post somewhere to show us how to produce the graphs such as in figure 2a, 2b, figure 3..., that would be nice, i always wanted to do that but don't know what tools to use and what's the procedure.)
Reply
2014-Jun-08, 01:59:35,
#4
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
I recommend publication as well.
Reply
2014-Jun-08, 02:40:46, (This post was last modified: 2014-Jun-08, 02:43:48 by Meng Hu.)
#5
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
Question.

In your article, you refer to Everson & Millsap...

Quote:used pathway analysis to show socioeconomic status only had a minor direct association with 1995 SAT scores but acted indirectly through high school achievement and “extracurricular activities” like “computer experiences” and “literature experiences.”

Is it really that article here ?
http://research.collegeboard.org/sites/d...scores.pdf

The kind of stuff they use must be in figures 2/3, no ? That's what I was afraid of. They commit the fallacy of ignoring "model equivalency". They shouldn't do that. They do not even cite MacCallum. So I'll do it.

MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure models. Psychological Bulletin, 114, 185–199.

If you want, you can add this reference to your article.

Concerning Sackett (2012) it's another disappointment. Here's what they say...

Quote:In the 2006 national population of test takers, the correlation between SES and composite SAT score was .46. Therefore, 21.2% of variance in SAT scores is shared with SES, as measured here as a composite of mother’s education, father’s education, and parental income. Thus, SAT scores are by no means isomorphic with SES, although the source of the SES-SAT relationship is likely due to some combination of educational opportunity, school quality, peer effects, and other social factors.

And here's what the same Sackett (2008) said :

Quote:High-Stakes Testing in Higher Education and Employment: Appraising the Evidence for Validity and Fairness (Sackett 2008)

Prototypically, admissions tests correlate about .35 with first-year grade point average (GPA), and employment tests correlate about .35 with job training performance and about .25 with performance on the job. One reaction to these findings is to square these correlations to obtain the variance accounted for by the test (.25 accounts for 6.25%; .35 accounts for 12.25%) and to question the appropriateness of giving tests substantial weight in selection or admissions decisions given these small values (e.g., Sternberg, Wagner, Williams, & Horvath, 1995; Vasquez & Jones, 2006).

One response to this reaction is to note that even if the values above were accurate (and we make the case below that they are, in fact, substantial underestimates), correlations of such magnitude are of more value than critics recognize. As long ago as 1928, Hull criticized the small percentage of variance accounted for by commonly used tests. In response, a number of scholars developed alternate metrics designed to be more readily interpretable than “percentage of variance accounted for” (Lawshe, Bolda, & Auclair, 1958; Taylor & Russell, 1939). Lawshe et al. (1958) tabled the percentage of test takers in each test score quintile (e.g., top 20%, next 20%, etc.) who met a set standard of success (e.g., being an above-average performer on the job or in school). A test correlating .30 with performance can be expected to result in 67% of those in the top test quintile being above-average performers (i.e., 2 to 1 odds of success) and 33% of those in the bottom quintile being above-average performers (i.e., 1 to 2 odds of success). Converting correlations to differences in odds of success results both in a readily interpretable metric and in a positive picture of the value of a test that “only” accounts for 9% of the variance in performance. Subsequent researchers have developed more elaborate models of test utility (e.g., Boudreau & Rynes, 1985; Brogden, 1946, 1949; Cronbach & Gleser, 1965; Murphy, 1986) that make similar points about the substantial value of tests with validities of the magnitude commonly observed. In short, there is a long history of expressing the value of a test in a metric more readily interpretable than percentage of variance accounted for.

Has he shifted his mind ? I don't believe that. It's more likely there's something wrong with him. The r² is a statistics that should never be used. He knew the problems with r² but still uses it. Well...

Concerning the Hampshire (2012) article, it has problems, and personally I would not cite it (although it's just my opinion, thus it worth little).
http://www.openpsych.net/forum/showthread.php?tid=38
Reply
2014-Jun-08, 15:27:37,
#6
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
I am in Leipzig, Germany right now, so am pressed for time and don't have access to academic literature.

Good submission. Very thorough paper.

Quote:Sackett et al estimated a correlation between SAT scores from 1995 to 1997 and socioeconomic status of 0.42, explaining 18% of variance. They defined socioeconomic status as a composite that equally weighted father’s education, mother’s education, and family annual income. Their meta- analysis of 55 standardized-test studies, less than half of which were for the SAT, gave the significant correlations of 0.255, 0.284, 0.223, and 0.186 for composite socioeconomic status, father’s education, mother’s education, and family income, respectively, without utilizing multiple linear regression. A similar study by Sackett et al (2012) determined that composite socioeconomic status explained 21.2% of 2006 composite-SAT variance.

This kind of compositing has been annoying me for some time. It is very arbitrary. As I see it, one should do either or both of the following two:

1) use the variables for the socialeconomic status and do a factor analysis (whichever type: PCA, PAF, maximum-likelihood, ...) and use the general factor. A lack of a general factor means that there is no such thing as the general socioeconomic status. However, one is universally found so this isn't an issue. :)

2) use multiple regression on the desired dependent variable to find the best way to weigh the variables for prediction.

The above methods may give the same results which makes for easy interpretation. :)
Reply
2014-Jun-11, 07:01:57,
#7
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
@menghu1001

“4. Variable x1, controlling for x2 and x3, has a large/weak/null effect on variable Y.
By variable, what is meant conventionally is its full effect, not its partial effect. That’s why the statement is a complete nonsense.”

“…regression with longitudinal data is the only way to prove the causality and by the same token to disentangle the pattern of direct and indirect effects. This is because causality deals with changes, that is, the impact of the changes in variables x1 and x2 on the changes in variable Y. But changes is only possible with repeated measures and only if the variables are allowed to change meaningfully over time…”

“…multiple regression should be used to test whether any variable can be weakened in its direct effect by other independent variables but not to test between them as if they were in competition with each other.”

Thanks for reviewing. I incorporated some of your corrections. I didn’t take out discussion of the suppression effect of race and parents’ education on parents’ income because that would obviously gut the paper, and you haven’t persuaded me that multiple regression is incapable of studying suppression effect. The longstanding statistical concept of suppression effect in regression can be found in the <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2819361/">scientific literature</a> and <a href="http://books.google.com/books?id=xUy-NNnSQIIC&pg=PA1102&lpg=PA1102&dq=%22suppression+effect%22+regression&source=bl&ots=SKCM4auZIS&sig=zPjhDZANYONtIyNbFyYopMgt-e4&hl=en&sa=X&ei=UPKTU9KBJMiVyATb3IK4BQ&ved=0CE0Q6AEwBTgK#v=onepage&q=%22suppression%20effect%22%20regression&f=false">textbooks</a>. I see that you have a strong preference for SEM and pathway analysis, but I’m not seeing a methodical disproof of suppression effect in multiple linear regression. I did attempt to test variable interaction effects. Apparently, I didn’t do it right, but if my preliminary results were any indication, then the interaction between race and parents’ education was much stronger than other interaction variables. This would have been just one more piece of evidence that parents’ income hardly matters. Perhaps your thinking is also influenced by the issue of multicollinearity, and I agree that is an important issue for underpowered studies. If income somehow has a large indirect effect despite its tiny direct effect, assuming your logic is sound, of which I am unconvinced, then the fact that I have made clear that my methodology was based on multiple regression will not discourage future research using SEM. I am innocent of false advertizing. I think I mostly avoided causal language. I edited my study to use words like “associated” and “predicted,” rather than “caused,” or “made.” All science is vulnerable to postmodernist critique. Even longitudinal observations are vulnerable to the fallacy of post hoc, ergo propter hoc. As you and I noted, SEM is vulnerable to confirmation bias or equivalent models. My use of multiple linear regression is too simplistic, unless you compare it to everything else done so far: the mere correlations cited by university presidents and media personalities and the published studies that assume SES is two parts education, one part income. My imperfect approach resulted in very high R^2 values.

I have noticed that certain methodologies (SEM, GWAS) gain cult-like followings and become gods that demand that we put no other gods before them. You want to make the perfect evidence the enemy of good evidence, but perfect evidence either doesn’t exist, or those who hold it have vested interests in hiding the truth. Everson and Millsap, of the College Board, also feel that multiple regression is outdated, and their use of SEM conveniently divided part of the effects of race into between-school differences, so as to “provide little support and comfort to those critics” who have “attacked high-stakes tests such as the SAT as biased, unfair, or discriminatory.” Thus, SEM let them shoehorn the data into a facile narrative. Racial injustice explains the gap, but it’s not their fault, of course. My study has value precisely because it is DIY with publicly available data, which gives me the freedom to test the assumptions of both the race-based criticisms of the SAT and the attempts to deflect racial controversy by the College Board.

Regarding R^2, I think your criticisms apply more to the studies by Schmidt et al than mine. I am comparing the R^2 values of different regressions to each other, not looking at one R^2 in isolation in order to make it seem small. Since my R^2 values are already so close to one, replacing them with R values would make little difference. However, it would make my presentation of the data inconsistent with the studies I cited. Also, adjusted R^2 is considered more appropriate for multiple linear regression than R^2. Is there such a thing as “adjusted R”?

“write a blog post somewhere to show us how to produce the graphs such as in figure 2a, 2b, figure 3...,”

All graphs were with Matlab using add-on commands.
x=[1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4];
y=[20 40 60 80 100 20 40 60 80 100 20 40 60 80 100 20 40 60 80 100];
z=[1.90E-30 1.80E-55 2.20E-58 2.30E-62 4.80E-62 3.20E-09 6.60E-06 0.053 0.34 0.41 7.80E-23 5.30E-12 0.052 0.083 9.40E-05 2.10E-60 4.30E-40 4.70E-17 2.00E-06 0.083]
tri=delaunay(x, y);
trimesh(tri, x, y, z);
set(gca, 'zscale', 'log')
campos([-6.2462,683.8065,-1.8871])

Figure 3 uses plotyy(x, y1, x, y) to plot the right vertical axis plot. I made the graphs discontinuous, as follows:
% education beta
x=[1 2 3 4 5];
y=[-1.17E-02 1.69E-02 7.59E-02 1.10E-01 1.38E-01];
x=[x,5];
y=[y,NaN];
x=[x,6 7 8 9 10];
y=[y,-1.17E-02 -1.37E-02 1.49E-02 2.81E-02 4.33E-02];
x=[x,10];
y=[y,NaN];
(etc.)
l1 = plot(x,y);
hold on;
Reply
2014-Jun-11, 21:17:30, (This post was last modified: 2014-Jun-11, 21:49:53 by Meng Hu.)
#8
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
I cannot access your link toward google books. However... MacKinnon et al. (2000) said stuff like that,

Quote:The signs and magnitudes of the τ (note : total effect in model with 2 var) and τ’ (note : direct effect in model with 3 var) parameters indicate whether or not the third variable operates as a suppressor. If the two parameters share the same sign, a τ’ estimate closer to zero than the τ estimate (a direct effect smaller than the total effect) indicates mediation or positive confounding, while a situation in which τ is closer to zero than τ’ (a direct effect larger than the total effect) indicates suppression (or inconsistent mediation or negative confounding, depending on the conceptual context of the analysis). In some cases of suppression, the τ and τ’ parameters may have opposite signs.

Hmm... well, I think I'm aware of that, but I'm not completely sure what's your point. See below.

Quote:I see that you have a strong preference for SEM and pathway analysis, but I’m not seeing a methodical disproof of suppression effect in multiple linear regression

MR functions the same way as SEM. the regress coeff is nothing more than the portion of total effect in SEM that is labeled "direct path". Therefore, there is no possibility to say which one has the strongest effect. As I said in my article, the most relevant part is the sentence that says that the regress coeff depends solely on the strength of the direct effect. I'm not saying MR is useless, because if the addition of predictor does not weaken the variable, say, income, then you are confident that most of its effect is direct. Thus, you don't need to posit assumptions regarding the causal pathways among the predictors, i.e., the kind of thing you're doing when using SEM.

Quote:I see that you have a strong preference for SEM and pathway analysis

Not exactly. I'm not a proponent of SEM+cross-sectional data, but a proponent of SEM+longitudinal data. Unfortunately, the latter case is the exception, not the rule. Most people who use SEM don't know what they are doing. Everson & Millsap is another illustration of what I say.

Quote:If income somehow has a large indirect effect despite its tiny direct effect, assuming your logic is sound, of which I am unconvinced, then the fact that I have made clear that my methodology was based on multiple regression will not discourage future research using SEM.

If you're not convinced, you must explain me why and where I am wrong. Furthermore, SEM and MR have more similarity than what you seem to believe. SEM disentangles direct and indirect effects, unlike MR. Because of that, in SEM you must make assumptions about which causal model is more likely than others, based either on theory or data (if you use longitudinal data). My point is that MR is just like SEM, with the pattern of indirect effect removed. And my question is : why would you think the regress. coeff. are necessarily comparable ? MR can give you, indeed, an indication of a suppression effect, based on what MacKinnon et al. said above. I already knew that, but the most relevant question should be : what's the total effect of each of your independent var. ? Can you calculate them ? My point is that you can compare the predictors only if you can calculate for them their total effect.

I have made modification to my articles, i'm not sure if you have read it yet but I said : " What is even more insidious is that the direct and indirect effects can be of opposite signs (Xinshu, 2010). Thus, in multiple regression, not only it is impossible to judge the total effect of any variables but also it is impossible to judge the direction of the correlation." the authors made it clear that direct and indirect effects are not necessarily complementary but there can be competitive mediation, which happens when direct and indirect paths have opposite signs.

I know it's a difficult question since the researchers will not recognize it. I have email lot of authors (~20) but only 3 responded. 2 of them thanked me, the other did not understand what I say, "there is no controversy", "I have not compared the predictors among them" stuff like that. It's been 3-4 days now. This is not encouraging, especially because my stats in my blog is extremely low, which mean they have not even bothered to click on the link toward my article.

There is a last comment I received from Paul Sackett, although he did not read my articles that I suggested him to do. He only says he agrees with me that we must not use r² as measure of effect size, but we can report it to say it represent the "amount of variance" etc., and nothing more.

Quote:Since my R^2 values are already so close to one, replacing them with R values would make little difference.

I saw that already, but you haven't said explicitly you did not consider them as measure of effect size, because most people see them as effect size. Each time I see someone reporting r² I expect him to interpret it the wrong way unless he made it very explicity that r² is not an effect size and should not replace r.

Quote:As you and I noted, SEM is vulnerable to confirmation bias or equivalent models.

Agreed. But I think it's even worse than that. If you use SEM, it takes you only few minutes to spot that fallacy. Simply by reversing the arrows or drawing a "covariance path" instead of "causation path" between the independent var., will immediately let you know the full extent of the fallacy. Researchers must know that. I can't believe they have never noticed it before. Thus, my assumption is that they are doing something they know is not robust to equivalent models. That makes me really upset. Lot of studies using MR and SEM is garbage.

Quote:All graphs were with Matlab using add-on commands.

Thanks, I'll try that. I hope Matlab is free, because if not...

Emil and duxide, if you can read this message...

i would like to request something important. I don't have much free time, probably you don't either. Each time I review some articles, when the author makes changes, he has never precised WHICH part of the article is modified or what has been added. I must constantly search for the addition and modification, which causes me to re-read the article entirely once again. I prefer not to waste my time searching for this, for that...

So, I suggest either 2 options :

1. the author copy-past in the forum which portion is added or modified.
2. the author marks in color (blue or red) the portion that is added or modified.

What do you think ?
Reply
2014-Jun-11, 22:42:00,
#9
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
I am not reviewing the manuscript, but is it OK to make a simple suggestion? Please spell out the full name before using the SAT and ACT. This is of minor importance, but I have run into confusion with the SAT because it is the same acronym used for the Stanford Achievement Test.
Reply
2014-Jun-11, 23:33:21,
#10
RE: [ODP] - Parents’ Income is a Poor Predictor of SAT Score
@menghu1001

“Therefore, there is no possibility to say which one has the strongest effect.”
“If you're not convinced, you must explain me why and where I am wrong.”

You think, (correct me if I’m wrong) that R^2 underestimates the true explanatory power of a model, and that MR underestimates the true explanatory power of a variable because it only includes direct effects. So, if my model can achieve an R^2 of 0.942, then adding in those indirect effects could add up to 0.058 to the coefficient of determination, and you want me to think that such a miniscule change would so profoundly transform my model that, in its current form, “there is no possibility to say which [variable] has the strongest effect.” I would say that my study is evidence that you misunderstand MR.

“why would you think the regress. coeff. are necessarily comparable ?”

Since they are not comparable, any semblance to the real world is just a coincidence. Here is a list of coincidences found in my data:

1. Asian race improves math score.
2. Asian race somewhat hurts verbal score.
3. The racial variable always matches the national reports of racial gaps (i.e. always positive except for Asian race with verbal scores).
4. A study that includes all these races, but singles out Black people as the lowest group and Asians as the highest group would be expected to show small effects for Asian race because they score closer to the majority race.
5. Affirmative action exists, and it allows certain minorities to achieve levels of income and especially levels of education beyond what their SAT scores would predict. Therefore, a racial variable should matter more at higher incomes and especially at higher education levels.
6. The higher influence of education compared to income matches Cohen’s d advantage magnitudes.
7. The higher influence of education matches expectations based on high estimated hereditary influence on the SAT and high bivariate overlap with GCTA.
8. Having at least a high school education is such a powerful norm in the US that it is extremely common, so much so that the designation of high school graduate should be pedestrian and uninformative.
9. Students who take the SAT in ACT states probably tend to be smarter than those in ACT states who don’t.
10. SAT participation rates are declining over time.
11. More populated states tend to be SAT states.
12. The Flynn effect is thought to have ceased in the US.

These are all just coincidences, and I’m sure that you could find just as many notions suggested by my data that are contradicted by real-world observations. If not, my study might prove magical forces are at play.

“I hope Matlab is free”

I don’t think so. Torrents are free but illegal and, therefore, strictly forbidden!

@csdunkel

“Please spell out the full name before using the SAT and ACT”

I don’t think they officially stand for anything anymore. The SAT stood for Scholastic Aptitude Test until 1990, then Scholastic Assessment Test until 1993, when it became a pseudo-acronym. ACT stood for American College Testing, but I think it is a pseudo-acronym, too.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)