Building on what John wrote yesterday, US students had significant, on average, lower test scores in the past two years.
National test results released on Thursday showed in stark terms the pandemic’s devastating effects on American schoolchildren, with the performance of 9-year-olds in math and reading dropping to the levels from two decades ago.
This year, for the first time since the National Assessment of Educational Progress tests began tracking student achievement in the 1970s, 9-year-olds lost ground in math, and scores in reading fell by the largest margin in more than 30 years.
I am not an education researcher, but I am a reasonably competent causal inference researcher that can bring some general observations to this problem if the hypothesis is that school closures are to be assigned dominant responsibility. I have a strong belief that any one factor will only explain a chunk of the entire variance.
The TLDR: This is one hell of a nasty inference problem with significant confounding hovering in the background.
If we think that school closures are driving test score loss, we would expect some of the following to be true:
- Areas with more closures should have more loss “Dose-response”
- Heterogeneity needs a strong theoretical explanation
This is tough to show:
Oddly, reading scores in cities, many of which had long-lasting school closures, did not drop at all during the pandemic.
And scores in the Northeast, where school closures were common, dropped only a bit more than in the South, where closures were not. https://t.co/zewMxHMU6e
— James Surowiecki (@JamesSurowiecki) September 1, 2022
At this point the sociologists in my life are screaming in my head with the note that the most marginalized groups of a population are usually the ones who get kicked hardest first and get kicked longest by a negative shock. The methodologists in my life are now screaming that school closures are not exogenous. They don’t happen by chance. School close as a function of both local politics and local disease incidence or disease consequence. Schools closing can happen because there is massive, ongoing spread in the community killing parents, grandparents, teachers and community members. Closure may be highly correlated with systemic trauma.
Untangling the causal pathway with anything solid means dealing with massive confounding. There are designs that can get local estimates of the effects of school closures on educational outcomes but studies that use non-random choice of districts within one part of one state will have a hard time generalizing to the nation. Super sophisticated techniques can and will (eventually) get around the challenges of someone running a simple regression making huge claims.
Argiope
I love that you have both sociologists and methodologists screaming in your head, David. We could probably all use more of that. Yet, the noise…..
Another Scott
Yup, drawing “obvious” conclusions is deadly. Look at how the federal response changed when some monster decided that SARS-CoV-2 was killing Democrats so that was good…
Thanks for the reminder.
Cheers,
Scott.
Brit in Chicago
Thanks for your cautious scientific approach to this issue (an attempt to get actual useful knowledge). It’s all too rare.
Anonymous At Work
If you have both sociologists and methodologists screaming in your head, you need help. Go sit in a theoretical physics labs until the screaming stops, at least once a day.
Especially when I mention that data collection issues and quality issues are known but not quantifiable generally, much less under pandemic conditions. Therefore, data at county, school district, and state levels are so bad, the only solution is to nuke the site from orbit.
Suzanne
I have no doubt that school closures were damaging to educational outcomes. There was plenty of data before the pandemic indicating that online education was just not as successful at transmitting information as in-person education. That, to me, isn’t the question. The question should be whether or not school closures resulted in better health outcomes for a community.
Bill Hicks
I’m trying to get the gist of your post, but your jargon makes it more difficult. Are you trying to say that pinning any loss in NAEP scores on the pandemic and school closures is not warranted, or that it is probably one of many factors, or it is probably not related? Mostly it sounds like the usual, correlation is not causation and observational studies (which are probably the only possibility in this case) can not prove causality. If so, boring. Even well done manipulative studies can’t prove causation (https://en.wikipedia.org/wiki/Karl_Popper).
David Anderson
@Bill Hicks: Correlation is not causation —- learning loss happened, NAEP is pretty damn close to gold standard, but the factors that caused the learning loss are going to be hard to disentangle so simplistic statements of “Closing schools ==== learning loss” is probably overstating the evidence.
BradF
Oh my.
Today’s WSJ editorial page needs a slice of what you are serving, David. Bigly.
Randi Weingarten Flunks the Pandemic – WSJ
Lobo
Thank you for this! But it so easy to look at something and say A==>B. Using this logic we could equally say Covid ==> low test scores. People don’t like “it is complicated.” Sigh!
Llelldorin
So… a room with a desk and a really big whiteboard?
Bumper
Thanks for writing this. There seems to be an anti-union push here, blaming teachers’ unions for drops in test scores. In addition to the WSJ report noted above, USA Today published a piece by Ingrid Jacques a few days ago, which showed up in my newsfeed as an news article, not as an opinion. There needs to be pushback.
Bill Arnold
@Lobo:
Hm. Are the raw data loosely consistent with such a causal hypothesis?
A friend’s son, high school age, had (and maybe still has; haven’t asked recently) cognitive/memory dysfunction after a case of COVID-19. No pneumonia. Bad enough to interfere with school work.
emmyelle
I guess another response to these findings would be: Wow, reading scores dropped due to circumstances. Let’s make some plans for helping these students over the next year.
Bill Hicks
Thank you David, I still think closing schools was a cause of the phenomenon mostly because it was pretty damn obvious. Whether closing schools was a proximate, ultimate, direct, indirect etc., (I prefer indirect) cause, it was almost definitely (can’t prove things true) a cause. Causation acts at multiple levels. In my bio classes I teach about how causation can be explained at various levels from the molecular to the evolutionary and they are complementary/mutualistic. They all must be true if you are truly close to understanding. I find it hard to believe that these NAEP scores can be explained in the absence of the epidemic and closing of schools.
Sorry if I am being cranky, I am probably misunderstanding you. I felt that what you wrote minimized the effect of closing schools on learning in this instance. Yesterday, in the linked post, there was a lot of scoffing about standardized test scores in the comments and I interjected that it was unwarranted. It scared me how easily even non-republicans will throw science/data on the dustbin. I am glad you have unambiguously supported NAEP data and science.
Edith
@Suzanne: This is where I am too, and I don’t think we’ll know the full answer for awhile, if ever. Take for instance, that the South experienced slightly less learning loss. They also had massively higher death rates. What part of that higher death rate is due to school closures versus lower vaccination rates versus more relaxed community standards on social distancing versus demographic and socioeconomic differences. I’m not sure how you’d ever isolate the different factors.
If I had to pick the total cultural package today, I’d go with the more school closings/lower death rate cultural package. There was about a two point difference in learning loss. I think I read the each point is nine weeks of instruction. So students in the South lost about a year and a half, and students in the Northeast lost about a year and a half. Personally, I’m ok with that half year difference if it means the difference between MA and TX death rates, and most likely difference in number of infections given that upwards of 5 percent of kids develop post infection health issues.
Part of that is that reading and math scores have risen so consistently that even with these losses, reading scores are where they were in the mid-nineties. As a late GenXer, it’s feels a little weird to have people talking about this terrible disaster of having kids be better at reading and significantly better at math than my peers and I were. To be honest, given the severity of the pandemic, I think it’s pretty good that kids on are track to have school outcomes as good or better than half of the current workforce.
oldster
@Bill Hicks:
“I still think closing schools was a cause of the phenomenon mostly because it was pretty damn obvious….”
I hope you are troubled by the data that Surowiecki points to, which constitutes a major objection to your hypothesis: places with *more* school closures had sometimes the *same* amount of learning loss, and sometimes even *less* learning loss.
That’s not how things should look if the “pretty damn obvious” causal hypothesis is true.