Achievement Gap – Random Critical Analysis

No, US school funding is actually somewhat progressive.

June 22, 2017November 17, 2018by rcafdm10 Comments

Note: I have already touched on these issues in a much longer, much broader post on education and on twitter at some length, however I thought it’d be useful to zoom in on this issue and marshal the evidence in one place for future reference.

It is still commonly supposed by much of the public that school funding is terribly unequal due to reliance on local funding mechanisms (especially property tax). Although there were once modest inequalities associated with local income levels (several decades earlier), this information is generally wildly out of date today. Within the vast majority of states districts with less advantaged students (read: higher poverty, lower income, fewer parents with college degrees, minority, etc) actually spend at least as much money per pupil (often more), both overall and in the narrower instructional expenditure category, and where there are inequalities these differences are usually quite modest and fleeting.

Though school funding is still significantly a local affair in most states there is substantial progressive redistribution of state and federal funds that effectively offset these potential inequalities (and then some). Some districts may choose to spend more controlling for income/wealth (tax effort) and there is some variance (mostly poorly explained by any SES measure), so that malcontents can always find isolated examples to complain about, but various formulas employed at the state and federal level sets a floor and effectively acts to prevent there being substantial positive correlations between school spending and district median income (or low poverty rates, percent minority, school free lunch percentage, and so on …. this holds across multiple measures).

My response to the NYTimes article on school districts, test scores, and income.

May 9, 2016November 17, 2018by rcafdm13 Comments

On April 29th the New York Times posted a nominally data driven article on school districts, test scores, and socioeconomic status. Though it contained some useful data, the analysis was terribly misleading and it excluded a tremendous amount of pertinent information. Many progressives took the article as proof that the system is “rigged”.

The NYTimes did not help matters by conflating the measures of socioeconomic status (SES) with income. Although every one of their plots used a composite SES measure on the x-axis, the article itself and various annotations give a strong impression that money/income/wealth are the primary drivers of this:

The words income, economic, wealth, money, rich, poor, and other related words were littered liberally throughout the article. Not a single mention was made of other predictors or even of the composition of the SES index they used, save for an easy to miss footnote at the end of the article.

The SES measure they used was defined in the SEDA archive as:

the first principal component factor score of the following measures: median income, percent with a bachelor’s degree or higher, poverty rate, SNAP rate, single mother headed household rate, and unemployment rate

[emphasis mine]

These non-economic dimensions actually exert significant influence on the correlations, are not directly tied to income/wealth/etc, and show marked racial/ethnic differences even at the same income level (e.g., single motherhood rates are much higher in the black community at any given level of income)

Rather than focus too much on what is wrong with this specific article (this sort of article is practically a genre unto itself these days), I will instead systematically address misperceptions here and attempt to shed more light on the nature and underlying causes of these patterns. I will argue that (1) these gaps are mostly genetic (2) they generally have little to do with systematic differences in parental economics (3) they have even less to do with the school systems themselves (4) these patterns are not unique to the US.

I will use some data from the Stanford Education Data Archive (SEDA), the same used by the NYTimes article, to help make some of my points but, unlike some of my other blog posts, I will try to cover each point in just enough depth to convey the gist of it (linking out for more in-depth analysis for those that are interested in the particular point). I will also bring together good deal of supporting evidence that is buried in different academic articles, government databases, think-tank research, and so on.

No, the SAT doesn’t just “measure income”

November 25, 2015November 17, 2018by rcafdm8 Comments

I started this post to refute some specific arguments, but I changed my mind midstream and decided to add a lot more material than I initially envisioned. This is best viewed as being akin to a FAQ (Frequently Refuted Objections – FRO?) relating to the standardized tests and their use in higher education.

table of contents

1) SAT and income are not perfectly correlated
2) Income correlates well with lots of tests
3) mean GPA, academic rigor, and the significance HSGPA varies with SES
4) Even controlling for SES, different racial groups see different average scores
5) SES & B-W cognitive gaps start very young and keep growing
6) The B-W test score gaps are poorly explained by schools
7) The differences in “school quality” are generally overstated
8) The SAT’s association with family income is almost entirely mediated by family education
8) SES does not significantly mediate the predictive power of the SAT
9) Some have misconstrued the Univ. of Calif. data to argue otherwise
10) Nor are these tests biased against minorities
11) The predictive power of the SAT is vastly under-estimated
12) The strength of this prediction does not weaken past freshman year
13) Holding HSGPA constant the SAT offers significant incremental validity
14) The SAT is well correlated with IQ tests
15) Lumosity’s cognitive tests show strong correlations with SAT across universities
16) SAT test prep has little effect
17) Test prep rates do not vary all that much
18) SATs predict graduation rates between schools too
19) When schools systematically discount standardized tests the effects are very obvious
20) Heritability explains a great deal of these systematic SES relationships

**SAT and income are not perfectly correlated**

The SAT is certainly modestly correlated with parental income, but it is simply not true that the SAT is nothing more than a measure of family income.

I will briefly plot the 2011 SAT reading scores by income level to illustrate that the r**2 is considerably less than one.

There is significant overlap across the entire income distribution:

Box and whisker plot of simulated test scores

Simulated distribution by actual test taker counts

On the effects of wealth on the B-W gaps, a response to questions posed by a commenter

June 28, 2015November 17, 2018by rcafdm3 Comments

Max, a commenter, asked:

Could you do an analysis on racial differences in educational outcomes after controlling for parental education, parental occupation, household wealth, neighborhood wealth, neighborhood education, single parent status, native language etc.? I’ve seen you control for family income and parental education (and occasionally both), but I’ve never seen you control for more beyond that (perhaps I’ve missed something!). In Chapter 16 of ~~Affirmative Action for the Rich~~The Future of Affirmative Action, Dalton Conley of New York University used the Panel Study of Income Dynamics to show that parental wealth (not income) and parental education are the best predictors of college completion, which means that they may also be good predictors of other educational outcomes. He also discussed the data showing that racial wealth gaps are much larger than racial income gaps, which implies that wealth could account for a larger portion of the achievement gap than income. Could you do a similar analysis for IQ? The reason I’m asking for all this is that Carnevale and Strohl control for all of these factors and are left with a very small race effect: http://www.tcf.org/assets/downloads/tcf-CarnevaleStrivers.pdf

I have heard this bit about wealth before. I am deeply skeptical that wealth can mediate much, if any, of the B-W gap.

Before I dig into this though, let’s take a very brief look at some of the studies Max cited (see here and here) :

On sex differences within California public schools

June 17, 2015November 17, 2018by rcafdmComments

A year or two ago I read an article which demonstrated that sex differences in math and reading achievement are inversely related within and across countries on the PISA tests. The smaller the male-female math gap, the larger the verbal gap and vice versa. This tends to support the view that there are innate underlying differences in average abilities and interests between the sexes that strongly influence these patterns.

On educational attainment rates and income as a causative factor

June 13, 2015June 14, 2015by rcafdm3 Comments

As I mentioned to Robert VerBruggen in his latest piece on educational attainment and income, I do not believe that economic concerns are a major cause of differences in education attainment rates by income.

I previously analyzed this and related issues using the ELS:2002 data, but I decided to extend my analysis with NLSY97 and clarify my views, now that I have marshaled a fair amount of data to support my arguments.

First

Although the IQ (ASVAB) is an excellent predictor and generally mediates these differences fairly well, there are other systematic differences that are not fully accounted for when you control for IQ. High SES people, whether measured by income or educational attainment, typically have higher GPAs even with the same test scores.

(these differences would likely to be larger still if I did this as a composite SES index using education, income, occupational prestige, etc)

IQ, test scores, GPA, income, and related correlations from NLSY97

June 11, 2015June 13, 2015by rcafdm1 Comment

Diving into the NLSY97 dataset a bit right now and I thought I’d share some plots pertaining to IQ/achievement tests, income, GPA, educational attainment, and more. Nothing here is particularly novel, if you’ve looked for this sort of data before, but sometimes it’s nice to have additional independent analysis or alternative presentations of the data.

High school GPA-test score relationships

HS GPA by ASVAB in percentiles (IQ test)

HS GPA by PIAT (Peabody Individual Achievement Test, IQ test admin. in early childhood)

Understanding socioeconomic mobility

June 7, 2015June 11, 2015by rcafdm4 Comments

Although it may not seem like it at first blush, given the apparently modest correlations, the socioeconomic figures that I blogged about earlier largely agree with the published data on economic mobility.

They are measuring income (or earnings) whereas I am measuring with the ELS SES index (which is an equally weighted average of the respondents own earnings as of 2011, educational attainment, and occupational prestige), but the systematic income differences between classes (however measured) are almost certainly virtually fully mediated by this more comprehensive SES measure.

ELS data, students 2011 SES by 2002 SES of parents

The correlation between parent SES and child SES is 0.35. This may not sound like much, but if you bin child SES and parent SES into quintiles the mobility estimates are very similar to the widely publicized economic mobility estimates.

The average person born at the top of the SES distribution has little chance of ending up at the bottom and vice versa, but there is clearly a great deal of mobility that happens amongst less extreme groupings.

[Note: I didn’t correct these figures for oversampling, so I won’t claim they’re a perfect representation of reality, but they are generally pretty close in practice and they still are good for illustrative purposes.]

Mobility delta (child SES – parent SES) by parent SES

On average, as compared to their parents, high SES people are downward mobile and low SES people are upward mobile. This may seem counter-intuitive to some, but this makes sense because r < 1, i.e., there is non-trivial mobility, and this must be true for there to be meaningful relative mobility. Of course, just because there is mobility doesn’t imply that all people have an equal chance of ending up in any place in the SES distribution.

As I mentioned in my last post, there are other differences between groups besides just propensity to end up in particular SES bins and many of these differences are highly predictive of mobility — see test scores, HS GPA, etc. Indeed, they almost fully mediate outcomes, so talking in terms of “mobility” here can be a bit misleading because relative starting position (parent SES) tells you relatively little about what is likely once you have better information (e.g., test scores, HS GPA, etc).

Predicting economic mobility from 10th grade test scores

June 5, 2015June 6, 2015by rcafdm3 Comments

In my last post I briefly touched upon economic mobility vis-a-vis the link between test scores and subsequent adult incomes. Because these individuals were still pretty young, just a few years out of college (if they graduated), the earnings correlations were weaker than one might have expected. Since then I discovered an interesting continuous SES variable (F3SES) in the ELS:2002 data set that is probably a better measure of future earnings or mobility.

F3SES is the average of 3 inputs (2011 earnings from employment, the prestige score associated with the respondents current/most recent job, and educational attainment), each of which is standardized to a mean of 0 and a standard deviation of 1 prior to averaging.

Data users should note that, as of the third follow-up, socioeconomic status may be less-than fully stable for some third follow-up respondents, e.g., respondents with graduate-level education who are just beginning or have yet to begin their careers. Users should also note that F3SES does not account for the income, occupation, or education of the respondents spouse/partner, and therefore may not be fully indicative of household socioeconomic status as of the third follow-up.

NOTE: While the two versions of the BY family SES composites (BYSES1 and BYSES2) were created by differential assignment of prestige scores based on the 16-category BY occupation variables, F3SES is created by assigning prestige scores based on the 2-digit ONET code associated with the respondents current/most recent job as of the third follow-up.

While I am sure I could derive my own formula to produce a similar composite score, I’ll just use theirs for the time being.

There is no statistically significance difference between blacks and whites here.

Asian SES is higher than white SES for most of the distribution, but that’s not statistically significant either.

Exploring ELS 2002 data

June 4, 2015June 6, 2015by rcafdm1 Comment

The NYTimes turned me onto a new data source in a recent article on college graduation rates by SES. They implied that college graduation rates are better predicted by “wealth” than by the students test scores (10th grade ELS scores taken in 2002).

Being both curious about the underlying data and somewhat skeptical of the particular claims (or, at least, its interpretation) I decided to investigate it for myself. Having done so now, I can tell you that it’s a pretty rich data set. Unfortunately, a few key data points (e.g., SAT scores, ~~HS GPA~~, etc) are censored or rounded/binned to protect anonymity, but there are still a lot of interesting data there to analyze.

Update (6/6/15):

As in my follow-up post on economic mobility, I realized that they actually provided 9th-12th grade high school GPA as a non-continuous variable in the publicly accessible file. I have updated my post to reflect this new information in a few places!

First point

The parent’s educational attainment is a much better predictor of both test scores and subsequent child educational attainment than economic measures…..

(Bachelor degree) Attainment rate by test scores, grouped by parent income levels