Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

Making Montssori

A popular new Montessori program in Detroit’s main district may expand into its own separate schools

PHOTO: Nick Hagen

Detroit’s main district is considering expanding its popular Montessori program, including possibly creating free-standing Montessori schools designed to draw students from around the city.

The possible changes could represent a major shift for the two-year-old program, which now operates in 14 classrooms in six schools.

Montessori parents have been on high alert in recent weeks. Told that changes are coming to the program, they’ve been worried that new Montessori schools would mean an end to existing programs.

“My son keeps asking ‘where am I going to school next year?’” parent Maria Koliantz told Chalkbeat last week.

Koliantz, who has two children in the Montessori program at Maybury Elementary School in Southwest Detroit, said a sense of brewing change has been affecting parents and teachers.

“I just keep trying to assure him,” she said of her son. “But … I hate that the uncertainty has affected him.”

But Superintendent Nikolai Vitti says there are no plans to move existing programs. Questioned by Koliantz at a school board community meeting Tuesday night, Vitti assured her the district would only add new programs, not close existing ones.

“We have no intention of discontinuing that program,” he said. “It’s a vehicle to recruit parents to the school system. I don’t think you’re going to see anything but expansion.”

Since his arrival in Detroit last spring, Vitti has talked about the need to give every school in the district a distinct identity, with some schools focusing on math and technology and others perhaps developing a focus on creative writing.

Vitti revealed Tuesday morning that the district is considering eventually creating three arts schools for children who’ve been identified as gifted or talented.

New Montessori schools are also on the table, he said. “The new schools will be announced by the end of March as we work towards ensuring that every school has a identifiable and distinct program to improve performance and enrollment.”

Freestanding Montessori schools could represent a new chapter for a program that was launched in Detroit two years ago as a hybrid system, with Montessori classrooms operating next to traditional classrooms in a handful schools.  

The program, which allows children to learn at their own pace in mixed-age classrooms, started in 2016 with classrooms serving pre-kindergarten and kindergarten students, as well as some students in grades 1-3 at Spain, Maybury and Edison elementary schools. The program more than doubled in size in 2017, adding classrooms in the first three schools and expanding into three more — Chrysler, Palmer Park and Vernor elementary schools.

But while the current structure at the six schools has been popular with some parents, it has also created some difficulties.

The Montessori program is run by a director, Nicola Turner, who hires teachers for the program, oversees their training, and supports them as they implement the Montessori curriculum. But those teachers also work for their school principals — a dynamic that can create complications.

In some schools, there has been tension between parents and teachers affiliated with the Montessori program and those connected to traditional classrooms. Since the Montessori programs tend to have more teachers and fewer students than traditional classrooms, that’s raised issues of fairness and equity.

The current setup has also created challenges aligning the Montessori curriculum with the structure and schedules of a traditional school. In an ideal Montessori classroom, for example, students would have an uninterrupted three-hour block to work on their core lessons, but that isn’t always possible in a school where many factors determine when students can have lunch, go to recess or take art and music classes.

Freestanding Montessori schools could avoid some of those problems — and potentially offer some advantages.

“We could do after-school programs that were Montessori-specific,” said Yolanda King, who has a son in the program at Spain Elementary and a younger child she hopes to enroll next year. Special classes like art, music and gym “could be more aligned to Montessori” in a freestanding school, she said, suggesting “yoga programs and whole food programs.”

Turner, the Montessori program director, declined to comment about the possible changes but an email she sent to parents this month indicates they were fairly divided about the prospect of freestanding schools.

Nearly half — 48 percent — said they preferred keeping Montessori classrooms in their current schools while 37 percent liked the idea of a Montessori school. About 15 percent did not indicate a preference.

Dan Yowell is among parents who’ve raised concerns that freestanding schools might feel removed from the rest of the district.

“We liked the fact that [Montessori] is accessible to people all over the city,” said Yowell, whose son is in the program at Spain.

A freestanding Montessori school “has a feeling that it’s more exclusive,” Yowell said. “I don’t want it to be perceived as something that only certain people can access.”  

Spain, in Detroit’s midtown neighborhood, is one of two schools with Montessori classrooms that has enough space to dramatically expand the program. The other one is the Palmer Park Academy, which is in northwest Detroit.

open questions

Segregation, struggling schools, ‘a larger vision’: What Councilman Mark Treyger is watching as NYC gets a new schools chief

PHOTO: Christina Veiga
City Councilman Mark Treyger is chair of the council's education committee.

As Mayor Bill de Blasio prepares to choose a new leader for the nation’s largest school system, no one is watching that decision more closely than Mark Treyger.

Treyger, a former history teacher who was recently named chairman of the city council’s education committee, will be responsible for holding the new schools chief accountable. In that role, the Brooklyn Democrat plans to support many of the de Blasio initiatives that the next chancellor will carry out — from expanded preschool to more social services in schools.

But Treyger also has some tough questions for the mayor and his yet-to-be-named schools chief. How do they plan to reduce school segregation? What is the mayor’s overarching vision for the school system? And must he choose the chancellor behind closed doors?

“I do believe that the best decisions are the ones where you involve critical stakeholders,” Treyger told Chalkbeat in a recent interview.

Below are some of the education issues that Treyger said he’ll be paying close attention to as de Blasio prepares to hand the reins of the school system over to a new chancellor.

1. What’s the larger vision for the school system?

Free pre-K has been de Blasio’s signature education accomplishment, but he’s also rolled out an assortment of lesser-known initiatives.

Many of them fall under the banner of “Equity and Excellence for All,” including efforts to make Advanced Placement classes available to all high-school students by 2021 and computer-science courses available to all students by 2025. Some critics have pointed out that many of those programs won’t be fully phased in until after de Blasio leaves. Others — including Treyger — wonder what they all add up to.

“It’s been a commendable beginning,” he said. “But I’m looking for a larger vision.”

On a practical level, Treyger also questioned whether the education department has laid the groundwork to roll out some of those initiatives. He said will work to make sure all schools have the infrastructure they need, such as reliable internet service and appropriate technology, to make sure they can offer courses like computer science.

“How can you have a conversation about computers,” he said, “when the lights don’t even work?”

2. Why not make the chancellor search public?

De Blasio has insisted that he won’t “crowdsource” the search for a new schools chief — despite calls for public input from a chorus of parents and experts.

Treyger thinks a compromise is possible: Let the mayor choose chancellor candidates, but then give the city council the power to vet the candidates during public hearings before signing off on the mayor’s pick.

“I believe that we should be open to moving towards a process where the city council has advise-and-consent power,” he said, adding that the legislature should consider altering the mayoral control law next year to give city lawmakers that power.

3. How serious is this administration about tackling school segregation?

School integration was not on de Blasio’s agenda when he came into office.

But after a grassroots movement of parents and educators called on the mayor to address the school system’s severe racial and socioeconomic segregation, he took some small steps in that direction. The education department released a “school diversity” plan last year, and has launched an integration-aimed admissions program at a few dozen schools and in one Manhattan district.

However, Treyger thinks the city can and must do more — including aligning school enrollment, zoning, and housing policies to work towards the same goal of integration.

“If we’re serious about addressing [segregation], we have to know the difference between managing the problem and actually solving it,” he said. “I think that we’ve seen, thus far, more management than actually solving.”

4. What’s next for the Renewal program?

The mayor’s $582 million “Renewal” program for struggling schools is at a crossroads.

De Blasio made a big bet that his administration could quickly rehabilitate 94 low-performing schools by giving them extra social services and academic support. But the program has achieved mixed results, and now the education department is planning to shutter eight Renewal schools next year — part of the largest round of school closures under de Blasio.

Meanwhile, another 21 schools that officials say have made significant progress will slowly transition out of the program.

Treyger’s first oversight hearing as education chairman, set for next week, will focus on the program. He has spent the last few weeks visiting schools in the program and says he wants to understand what the city’s future plans are for supporting those schools. And he wants to be sure that if struggling schools improve enough to leave the program, their extra support won’t suddenly be cut. (The education department has committed to maintaining the full budget allocation they receive through the city’s funding formula and extra social services for the 21 schools that are improving enough to leave the program.)

“We will not be happy,” he said, “if we learn that a school that is improving or turning things around — that its reward is a funding cut.”