If a child does not keep up with peers academically, summer school seems like an easy matter. Instead of forgetting what they have learned during the school year while on vacation for two months, they catch up and move on. Therefore, it was a surprise when one Rand Corporation study of summer school programs in five urban school districts found that this sensible solution… did not really solve the problem.
Rand’s study showed that summer school offered modest, short-term improvements in math results at best, but those improvements disappeared in the fall. Other measurements – performance in language art, student participation and overall grades – showed no meaningful connection to summer school. “The effects were pretty overwhelming,” said Megan Kuhfeld, a senior researcher at NWEA, a nonprofit organization for educational testing and research.
Overall, the summer school programs did not live up to their promises. But some subgroups benefited: the students who regularly attended the programs were better at navigating obstacles, such as retaining students.
It may never have been so urgent to make educational interventions as a summer school work for children. Two years into the pandemic, children across the country are lagging behind where they would have been academically if the pandemic had not happened. To help bridge the gap, education theories will need to adapt to the unique realities of actual children’s lives and the needs of families. If they do not, even the best ideas, with tons of evidence behind them, will not work in the real world.
Children learned a lot during the pandemic, Kuhfeld told me. The problem, she said, is that they do not learn as much or as fast as they did every year before the pandemic. At the national level, third grades in the fall of 2021 tested significantly below average, while third grades tested in the fall of 2019 in reading and math. The NWEA assessments showed that these declines also extend across third grades to eighth grades.
Most of the experts I spoke to said that the popular term “learning loss” is a misnomer – it’s not that kids have lost ground, they just don’t develop that fast. But the slower progression is real, and there are patterns in it. The effects were particularly pronounced among black, Hispanic, and American Indians and Native Alaska students.
In the NWEA data, the median percentile ranks for black third grades fell by 10 points in reading and 14 points in mathematics. For white third-graders, the median percentile ranks fell by exactly half of this (5 points in reading and 7 in math), while the median percentile ranks for Asian American third-graders fell by 3 points in both subjects.
In addition, there are signs of decline in attendance and graduation of high schools, something that could signal a broad sense of emotional separation from school. Which in turn could help explain slower learning – or exacerbate it, said Dan Goldhaber, director of the National Center for Longitudinal Data Analysis in Educational Research.
However, slow learning during the pandemic does not necessarily mean that children are doomed. In fact, other researchers like the Torrey Trust, a professor of learning technology at the University of Massachusetts Amherst, said that children actually learned a lot of things during the pandemic that they might not otherwise have learned. For many, virtual lessons meant more time with family, more skills with technology, and for some even better educational experiences, free from bullying.
The other good news: Research shows that the slower progress documented by these test results should be corrected with guidance in small groups. “It’s not rocket science,” said Thurston Domina, a professor of education policy and organizational management at the University of North Carolina at Chapel Hill. “You get the kids in small groups, and you can really give them customized instruction and focus on them.” When Matthew Kraft, professor of education and economics at Brown University, reviewed several meta-analyzes of the effectiveness of various pedagogical interventions in 2021, he found that tutoring in small groups had a significantly greater effect on students’ test scores than changes in class size, longer school days, or summer school-type programs.
But while it is relatively easy for researchers to conduct studies in classrooms or schools and find out which interventions yield the best results, it is difficult for educators to take those findings and put them to work across America. The evidence does not produce a solution – it just shows you how difficult it will be to create a far-reaching solution.
Example: these summer school studies. One of the biggest factors influencing the overall failure of summer school programs in the Rand analysis was that only about half of the children who attended one year did not return to the next – and some children did not even attend every day the first year. . The children who attended summer school usually in both years improved their math and language skills in ways that lasted the entire school year. But that group represented only about 35 percent of all children involved in the study.
So summer school works fine – if you can get kids to go. And it creates a completely different set of logistical complications that need to be studied and analyzed and implemented. It requires hiring the right teachers who have the motivation and the specific interest in teaching summer school, Kuhfeld said. It also requires long-term dedicated recruitment of children to the programs. Unlike regular school, students do not have to attend summer school, so getting them and their families to choose the educations means you have to build both interest and trust – which neither part is given. And all this requires money. “There is a big gap between what should work in theory and what works in practice,” Kuhfeld said.
This kind of effect is depressingly common. When the George W. Bush administration set up a program to gather evidence-based educational resources in 2002, education specialists told me they had hoped that this program – What Works Clearinghouse – would bridge the gap between academia and classrooms. They envisioned it as a way for teachers to gain better control over how to use evidence-based interventions in the classroom. “We thought we’d hit third-grade math and get an answer,” said Rachael Gabriel, a professor of literacy at the University of Connecticut.
But it never managed to be that simple.
In many cases, researchers I spoke to found that teachers – the people tasked with educating students and bringing up these test results – did not have much control over what interventions they could use and how. Those decisions were made higher up in the administration chain. A teacher might want to try something and not be allowed to. Or they may be excited to try something that was allowed but not get funding or staff or bus transportation to make it happen efficiently.
Making things work in a classroom is different from making things work in an entire district or an entire state or the entire country. That’s something Domina learned when California’s State Board of Education tried to mandate all eighth-graders to take and be tested in algebra. The idea was largely based on evidence, he said. Studies have shown that separating some children in elite mathematics and others in remedial mathematics served to expand inequality and narrow the future of children. Giving children higher expectations makes them do better. So an extension of access to algebra for all should have reduced the gap between rich and poor, white and black.
But it did not. In fact, the opposite happened. Domina sees economies of scale – especially staffing issues – at the heart of this failure. Offering algebra to everyone meant that schools needed many more algebra teachers, and that quickly. But there were only so many fully qualified, highly trained algebra teachers. Many children, especially those in lower-income schools, ended up with teachers who did not have as much experience and were not as effective at teaching the material, he said.
That story is particularly poignant now. Teaching in small groups can help students catch up on what they did not get a chance to learn during the pandemic. But guidance in small groups requires staff – and schools are one of many industries suffering from staff shortages. These are experts that Kraft is concerned about schools can create erroneous tutoring programs by using irregular volunteers or older students instead of committed staff.
Like students, schools do not necessarily function in a neutral, pre-pandemic state. “The biggest trend I have seen in the last 6-12 months is that schools are struggling to get the basics down. It is difficult to remain open, ”said Chase Nordengren, the primary research leader for effective teaching strategies at NWEA. He has seen many instances where federal funds, which may otherwise have been spent on staffing mentoring programs to reduce learning losses, were spent instead on things like better ventilation, personal protective equipment, and substitute teachers.
“I think guidance is a really promising initiative,” Goldhaber said. “But we have never tried to make guidance to the extent we try it today.” Because of that, he said, parents should encourage real-time evaluation and course correction to go along with these learning loss interventions. There should be tools in place to help teachers know when something is not working for their specific school, and allow them to make the kind of personal adjustments we know are necessary to make any intervention effective. But it again requires resources.
In the end, it is not children’s pandemic test results that really make researchers feel gloomy about the education of the future. Instead, it is the way education systems have been set up to fail these children. Schools have been running with limited resources and little room for change for at least the past decade, Domina said. “And now we’ve hit a crisis. And they’re not resilient.”