• Menu
  • Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

Before Header

  • About Us
  • Lexicon
  • Contact Us
  • Our Store
  • ↑
  • ↓
  • ←
  • →

Balloon Juice

Come for the politics, stay for the snark.

Tide comes in. Tide goes out. You can’t explain that.

Jesus, Mary, & Joseph how is that election even close?

One of our two political parties is a cult whose leader admires Vladimir Putin.

A democracy can’t function when people can’t distinguish facts from lies.

Balloon Juice, where there is always someone who will say you’re doing it wrong.

They traffic in fear. it is their only currency. if we are fearful, they are winning.

It’s easy to sit in safety and prescribe what other people should be doing.

75% of people clapping liked the show!

Text STOP to opt out of updates on war plans.

Since when do we limit our critiques to things we could do better ourselves?

Conservatism: there are people the law protects but does not bind and others who the law binds but does not protect.

Accountability, motherfuckers.

They punch you in the face and then start crying because their fist hurts.

Sadly, there is no cure for stupid.

Russian mouthpiece, go fuck yourself.

If you can’t control your emotions, someone else will.

Relentless negativity is not a sign that you are more realistic.

Tick tock motherfuckers!

A norm that restrains only one side really is not a norm – it is a trap.

Not loving this new fraud based economy.

The Supreme Court cannot be allowed to become the ultimate, unaccountable arbiter of everything.

Polls are now a reliable indicator of what corporate Republicans want us to think.

Speaking of republicans, is there a way for a political party to declare intellectual bankruptcy?

They are lying in pursuit of an agenda.

Mobile Menu

  • Seattle Meet-up Post
  • 2025 Activism
  • Targeted Political Fundraising
  • Donate with Venmo, Zelle & PayPal
  • Site Feedback
  • War in Ukraine
  • Submit Photos to On the Road
  • Politics
  • On The Road
  • Open Threads
  • Topics
  • COVID-19
  • Authors
  • About Us
  • Contact Us
  • Lexicon
  • Our Store
  • Politics
  • Open Threads
  • 2025 Activism
  • Garden Chats
  • On The Road
  • Targeted Fundraising!
You are here: Home / Anderson On Health Insurance / Causal inference and education loss during a pandemic

Causal inference and education loss during a pandemic

by David Anderson|  September 2, 20229:02 am| 16 Comments

This post is in: Anderson On Health Insurance

FacebookTweetEmail

Building on what John wrote yesterday, US students had significant, on average, lower test scores in the past two years.

National test results released on Thursday showed in stark terms the pandemic’s devastating effects on American schoolchildren, with the performance of 9-year-olds in math and reading dropping to the levels from two decades ago.

This year, for the first time since the National Assessment of Educational Progress tests began tracking student achievement in the 1970s, 9-year-olds lost ground in math, and scores in reading fell by the largest margin in more than 30 years.

I am not an education researcher, but I am a reasonably competent causal inference researcher that can bring some general observations to this problem if the hypothesis is that school closures are to be assigned dominant responsibility. I have a strong belief that any one factor will only explain a chunk of the entire variance.

The TLDR: This is one hell of a nasty inference problem with significant confounding hovering in the background.

If we think that school closures are driving test score loss, we would expect some of the following to be true:

  • Areas with more closures should have more loss “Dose-response”
  • Heterogeneity needs a strong theoretical explanation

This is tough to show:

 

Oddly, reading scores in cities, many of which had long-lasting school closures, did not drop at all during the pandemic.

And scores in the Northeast, where school closures were common, dropped only a bit more than in the South, where closures were not. https://t.co/zewMxHMU6e

— James Surowiecki (@JamesSurowiecki) September 1, 2022

 

At this point the sociologists in my life are screaming in my head with the note that the most marginalized groups of a population are usually the ones who get kicked hardest first and get kicked longest by a negative shock. The methodologists in my life are now screaming that school closures are not exogenous. They don’t happen by chance. School close as a function of both local politics and local disease incidence or disease consequence. Schools closing can happen because there is massive, ongoing spread in the community killing parents, grandparents, teachers and community members. Closure may be highly correlated with systemic trauma.

Untangling the causal pathway with anything solid means dealing with massive confounding. There are designs that can get local estimates of the effects of school closures on educational outcomes but studies that use non-random choice of districts within one part of one state will have a hard time generalizing to the nation. Super sophisticated techniques can and will (eventually) get around the challenges of someone running a simple regression making huge claims.

FacebookTweetEmail
Previous Post: « COVID-19 Coronavirus Updates: Thursday / Friday, Sept. 1-2
Next Post: TGIFriday Morning Open Thread: Some Speech! »

Reader Interactions

16Comments

  1. 1.

    Argiope

    September 2, 2022 at 9:08 am

    I love that you have both sociologists and methodologists screaming in your head, David.  We could probably all use more of that.  Yet, the noise…..

  2. 2.

    Another Scott

    September 2, 2022 at 9:10 am

    Yup, drawing “obvious” conclusions is deadly.  Look at how the federal response changed when some monster decided that SARS-CoV-2 was killing Democrats so that was good…

    Thanks for the reminder.

    Cheers,
    Scott.

  3. 3.

    Brit in Chicago

    September 2, 2022 at 9:18 am

    Thanks for your cautious scientific approach to this issue (an attempt to get actual useful knowledge). It’s all too rare.

  4. 4.

    Anonymous At Work

    September 2, 2022 at 9:26 am

    If you have both sociologists and methodologists screaming in your head, you need help.  Go sit in a theoretical physics labs until the screaming stops, at least once a day.

    Especially when I mention that data collection issues and quality issues are known but not quantifiable generally, much less under pandemic conditions.  Therefore, data at county, school district, and state levels are so bad, the only solution is to nuke the site from orbit.

  5. 5.

    Suzanne

    September 2, 2022 at 10:00 am

    I have no doubt that school closures were damaging to educational outcomes. There was plenty of data before the pandemic indicating that online education was just not as successful at transmitting information as in-person education. That, to me, isn’t the question. The question should be whether or not school closures resulted in better health outcomes for a community.

  6. 6.

    Bill Hicks

    September 2, 2022 at 10:20 am

    I’m trying to get the gist of your post, but your jargon makes it more difficult. Are you trying to say that pinning any loss in NAEP scores on the pandemic and school closures is not warranted, or that it is probably one of many factors, or it is probably not related? Mostly it sounds like the usual, correlation is not causation and observational studies (which are probably the only possibility in this case) can not prove causality. If so, boring. Even well done manipulative studies can’t prove causation (https://en.wikipedia.org/wiki/Karl_Popper).

  7. 7.

    David Anderson

    September 2, 2022 at 10:26 am

    @Bill Hicks: Correlation is not causation —- learning loss happened, NAEP is pretty damn close to gold standard, but the factors that caused the learning loss are going to be hard to  disentangle so simplistic statements of “Closing schools ==== learning loss” is probably overstating the evidence.

  8. 8.

    BradF

    September 2, 2022 at 10:42 am

    Oh my.

    Today’s WSJ editorial page needs a slice of what you are serving, David. Bigly.

    Randi Weingarten Flunks the Pandemic – WSJ

  9. 9.

    Lobo

    September 2, 2022 at 10:45 am

    Thank you for this!  But it so easy to look at something and say A==>B.  Using this logic we could equally say Covid ==> low test scores.  People don’t like “it is complicated.” Sigh!

  10. 10.

    Llelldorin

    September 2, 2022 at 11:17 am

    a theoretical physics labs

    So… a room with a desk and a really big whiteboard?

  11. 11.

    Bumper

    September 2, 2022 at 12:05 pm

    Thanks for writing this.  There seems to be an anti-union push here, blaming teachers’ unions for drops in test scores.  In addition to the WSJ report noted above, USA Today published a piece by Ingrid Jacques a few days ago, which showed up in my newsfeed as an news article, not as an opinion.  There needs to be pushback.

  12. 12.

    Bill Arnold

    September 2, 2022 at 12:09 pm

    @Lobo:

    Using this logic we could equally say Covid ==> low test scores.

    Hm. Are the raw data loosely consistent with such a causal hypothesis?
    A friend’s son, high school age, had (and maybe still has; haven’t asked recently) cognitive/memory dysfunction after a case of COVID-19. No pneumonia. Bad enough to interfere with school work.

  13. 13.

    emmyelle

    September 2, 2022 at 1:10 pm

    I guess another response to these findings would be: Wow, reading scores dropped due to circumstances. Let’s make some plans for helping these students over the next year.

  14. 14.

    Bill Hicks

    September 2, 2022 at 1:27 pm

    Thank you David, I still think closing schools was a cause of the phenomenon mostly because it was pretty damn obvious. Whether closing schools was a proximate, ultimate, direct, indirect etc., (I prefer indirect) cause, it was almost definitely (can’t prove things true) a cause. Causation acts at multiple levels. In my bio classes I teach about how causation can be explained at various levels from the molecular to the evolutionary and they are complementary/mutualistic. They all must be true if you are truly close to understanding. I find it hard to believe that these NAEP scores can be explained in the absence of the epidemic and closing of schools.

    Sorry if I am being cranky, I am probably misunderstanding you. I felt that what you wrote minimized the effect of closing schools on learning in this instance. Yesterday, in the linked post, there was a lot of scoffing about standardized test scores in the comments and I interjected that it was unwarranted. It scared me how easily even non-republicans will throw science/data on the dustbin. I am glad you have unambiguously supported NAEP data and science.

  15. 15.

    Edith

    September 2, 2022 at 1:42 pm

    @Suzanne: This is where I am too, and I don’t think we’ll know the full answer for awhile, if ever. Take for instance, that the South experienced slightly less learning loss. They also had massively higher death rates. What part of that higher death rate is due to school closures versus lower vaccination rates versus more relaxed community standards on social distancing versus demographic and socioeconomic differences. I’m not sure how you’d ever isolate the different factors.

    If I had to pick the total cultural package today, I’d go with the more school closings/lower death rate cultural package. There was about a two point difference in learning loss. I think I read the each point is nine weeks of instruction. So students in the South lost about a year and a half, and students in the Northeast lost about a year and a half. Personally, I’m ok with that half year difference if it means the difference between MA and TX death rates, and most likely difference in number of infections given that upwards of 5 percent of kids develop post infection health issues.

    Part of that is that reading and math scores have risen so consistently that even with these losses, reading scores are where they were in the mid-nineties. As a late GenXer, it’s feels a little weird to have people talking about this terrible disaster of having kids be better at reading and significantly better at math than my peers and I were. To be honest, given the severity of the pandemic, I think it’s pretty good that kids on are track to have school outcomes as good or better than half of the current workforce.

  16. 16.

    oldster

    September 2, 2022 at 5:06 pm

    @Bill Hicks:

    “I still think closing schools was a cause of the phenomenon mostly because it was pretty damn obvious….”

    I hope you are troubled by the data that Surowiecki points to, which constitutes a major objection to your hypothesis: places with *more* school closures had sometimes the *same* amount of learning loss, and sometimes even *less* learning loss.

    That’s not how things should look if the “pretty damn obvious” causal hypothesis is true.

Comments are closed.

Primary Sidebar

On The Road - Winter Wren - North of Quebec City (part 2 of 3) - Cap Tourmente and on the way to Tadoussac 2
Image by Winter Wren (5/13/25)

Recent Comments

  • Manyakitty on Tuesday Evening Open Thread (May 14, 2025 @ 1:52am)
  • Old Man Yelling at Clouds on Monday Morning Open Thread: Another Suggestion (May 14, 2025 @ 1:36am)
  • Jay on News of the Weird Open Thread (May 14, 2025 @ 1:35am)
  • Sister Inspired Revolver of Freedom on War for Ukraine Day 1,174: More Drone Swarms in the Small Hours of the Night (May 14, 2025 @ 1:19am)
  • AlaskaReader on War for Ukraine Day 1,174: More Drone Swarms in the Small Hours of the Night (May 14, 2025 @ 12:47am)

PA Supreme Court At Risk

Donate

Balloon Juice Posts

View by Topic
View by Author
View by Month & Year
View by Past Author

Featuring

Medium Cool
Artists in Our Midst
Authors in Our Midst
War in Ukraine
Donate to Razom for Ukraine

🎈Keep Balloon Juice Ad Free

Become a Balloon Juice Patreon
Donate with Venmo, Zelle or PayPal

Meetups

Upcoming Ohio Meetup May 17
5/11 Post about the May 17 Ohio Meetup

Calling All Jackals

Site Feedback
Nominate a Rotating Tag
Submit Photos to On the Road
Balloon Juice Anniversary (All Links)
Balloon Juice Anniversary (All Posts)
Fix Nyms with Apostrophes

Hands Off! – Denver, San Diego & Austin

Social Media

Balloon Juice
WaterGirl
TaMara
John Cole
DougJ (aka NYT Pitchbot)
Betty Cracker
Tom Levenson
David Anderson
Major Major Major Major
DougJ NYT Pitchbot
mistermix

Keeping Track

Legal Challenges (Lawfare)
Republicans Fleeing Town Halls (TPM)
21 Letters (to Borrow or Steal)
Search Donations from a Brand

PA Supreme Court At Risk

Donate

Site Footer

Come for the politics, stay for the snark.

  • Facebook
  • RSS
  • Twitter
  • YouTube
  • Comment Policy
  • Our Authors
  • Blogroll
  • Our Artists
  • Privacy Policy

Copyright © 2025 Dev Balloon Juice · All Rights Reserved · Powered by BizBudding Inc

Share this ArticleLike this article? Email it to a friend!

Email sent!