Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

choice and competition

It’s not just Detroit. Across Michigan, ‘active and aggressive’ competition imperils schools

PHOTO: Kimberly Hayes Taylor
Eric Lupher of the Citizen Research Council of Michigan, Benjamin Edmondson of Ypsilanti Community Schools, Randall Davis of Albion-Marshall School District and Scott Menzel of Washtenaw County Intermediate School District testify before the Michigan Civil Rights Commission

Detroit is not the only district struggling with lower enrollment and other challenges related to competition from charter schools and surrounding districts. On the other side of the state, similar forces led to the Albion school district’s demise.

After years of declining enrollment, falling revenue, poor student performance and school closures in the district, the Albion district in western Michigan faced a difficult problem: How to keep the district from dying. The city of Albion had a large number of students, but many of them travelled outside the city to attend school, forcing the Albion Community Schools district to merge with nearby Marshall Public Schools in July 2016.

Albion’s story was one of many shared Monday with state’s Civil Rights Commission, which held its first in a series of public hearings Monday in Ypsilanti to hear firsthand about issues confronting school districts. Representatives from public policy organizations, school districts, as well as parents, educators and advocates from Detroit and around the state shared stories of hardships and difficult decisions they face.

The commission is charged by the state’s constitution with investigating alleged discrimination. It launched the hearings this week after learning from education experts that state schools are in crisis. The goal of the hearings is to determine if minority students and those with special needs have faced discrimination in the state’s schools.

Albion’s story came from Randall Davis, superintendent of the Albion-Marshall School District, who told commissioners he blamed what happened to Albion schools — a district that had primarily served low-income, African-American students — on a law passed more than two decades ago that allowed students in Michigan to attend any school in any district that would take them.

“Schools of choice decimated the schools,” Davis said. “They had three or four of our contiguous districts that were driving into their district picking kids up. It was active and aggressive….I believe that is not the intention of schools of choice, but that’s what happened.”

The hearing was not intended to focus on competition from charter schools and between neighboring districts, but many of the people who testified came from traditional district schools or from policy organizations, so much of the testimony centered on the consequences of choice in Michigan.

Individuals also came forward to raise various concerns about equity in schools. The commission did not hear from charter school advocates but the commission plans to hold at least two other hearings.

“We also open it up to anyone to offer their opinions,” said Vicki Levengood, communications director for the Michigan Department of Civil Rights. “We encourage people on all sides to bring their messages to the hearings.”

The next hearing will be July 23 but a location has not yet been set, she said.

At the first hearing Monday, Benjamin Edmondson, Superintendent of Ypsilanti Community School District, painted the grim, poignant picture confronting him when he became the district’s school chief in 2015.

The Ypsilanti district lacked money to pay for services students needed such as social workers, homeless services, school safety officers, and washers and dryers. The district only managed to provide those services by partnering with the University of Michigan, Eastern Michigan University and community colleges.

“Those partnerships are critical,” he said. “I don’t know how we would survive without those entities buying into the district.”

Edmondson said his district had lost 300 students before he arrived, costing the district nearly $2.4 million. That forced him to scrap advanced placement classes and meant he couldn’t pay teachers enough to fully staff his classrooms. Graduation rates were low and the district was swimming in debt. The district’s challenges were compounded by the fact it competes with the nearby Plymouth-Canton Community Schools district, which enrolls more than 17,000 students. “We were David and Goliath,” he said.

The district also faces a “strong charter school presence,” he said, recalling a charter school representative who came vying to purchase a vacant district building. He said he felt threatened by the potential buyer’s ability to automatically take 300 students from the district.

“Here I am a new superintendent with a new school board and I just want to paint the story,” he said. “In 2015, we had declining enrollment, white flight, poverty, low expectations, low wages, high debts and priority schools, neighboring charter schools, and a state takeover threat for our schools.”

This year, Edmondson said the district is improving but still facing daunting challenges.

With population declines and fewer students in districts, even with consolidated districts, Michigan’s districts are too small, Eric Lupher of the Citizens Research Council of Michigan told commissioners. He said it means school districts from around the state are struggling because they are losing so many students to surrounding districts.

For example, he said Ypsilanti Community School District is continuing to bleed students to Ann Arbor, Plymouth-Canton and other districts. Ferndale schools are gaining almost 800 students from Oak Park and Detroit schools, but is losing about 350 students to districts further from city line such as Royal Oak and Berkley schools.

The issue isn’t limited just to southeastern Michigan, he said, pointing to Wyoming Public school district, about five miles from Grand Rapids, which has lost students to nearby Jenison and Grand Rapids.

“It’s a bigger issue, and it’s a lot bigger than just consolidation,” he said. “It’s the choice we’ve offered. I’m not here to speak ill of choice, but it’s creating issues we’re not dealing with.”

The eight commissioners listened intently through the six-hour hearing at the Eagle Crest Conference Center, Ann Arbor Marriott in Ypsilanti, occasionally asking questions.

Commissioner Jeffrey Sakwa at one point expressed sympathy for the superintendents. “You guys are in a tough place,” he said.

While Michigan once had nearly 600 school districts, Sakwa said, that number is shrinking.  

Sakwa blamed competing school districts as a primary reason for the changes.

“It’s like Burger King, McDonald’s, and Kentucky Fried Chicken all on the corner trying to steal everybody’s lunch every single day,” he said.

PHOTO: By Kimberly Hayes Taylor
Helen Moore

About 20 people from Detroit, Ypsilanti and Ann Arbor spoke during the public comments portion of the hearing. Among them was longtime Detroit education advocate Helen Moore. “Don’t play games with us,” she told commissioners over applause that sometimes drowned her words.

“You know the discrimination we have received as black people and our children. You know that during slavery it was against the law to read. This is what’s happening to our children now.”

 

Prize money

A million dollars, 570 hopefuls, and 15 winners: How a new competition aims to boost babies and toddlers

PHOTO: JGI/Jamie Grill | Getty Images
Boy displaying drawing.

A Colorado team is one of 15 winners to share in $1 million awarded by a Denver-based organization as part of a new contest recognizing innovative efforts benefitting children from birth to 3 years old.

The Boulder-based team will receive $80,000 for a project that helps little kids acquire language, thinking, and social-emotional skills using a cell phone app inside a stuffed animal.

Gary Community Investments, which gives grants and makes for-profit investments to benefit low-income children and families, announced the winners of the Early Childhood Innovation Prize on Tuesday afternoon. (Gary Community Investments, through the Piton Foundation, is a Chalkbeat funder.)

The Colorado team that won prize money developed a tool called MindScribe. It works like this. An adult slips a cell phone with a special application into the belly of a stuffed zebra. The app prompts the child to explain what they are doing or making and asks follow-up questions, such as “What happened next?” and “Why?”

MindScribe founder Layne Hubbard, a Ph.D. student in computer science at the University of Colorado Boulder, said her work as a teacher at Boulder’s Children’s House Preschool inspired the project.

“I thought back to storytelling and how powerfully the children’s original stories catalyzed growth, development, and connectedness,” she wrote via email. “I realized that I wanted to scale this opportunity to reach young children across diverse early childhood communities, especially those which are multilingual, low-income, or affected by trauma or disability.”

One little girl who stars in a MindScribe’s demonstration video describes her crayon drawing of a garden — and her fictional protagonist’s desire to change “boring weather” — to the MindScribe zebra for seven minutes.

But the girl, Mia, isn’t oblivious to the cell phone inside the paunchy stuffed animal. Instead, she’s delighted.

She explain how it works to her father, saying, “This is like the teacher but with a radio inside the teacher.”

Mindscribe, which is still in the pilot stage, began with three languages and is now available in 11.

The Early Childhood Innovation Prize, unveiled by Gary last fall, is distinctive because there are few contests that focus on very young children — despite a large body of evidence showing that high-quality care and education for this group yield significant financial and societal dividends.

Leaders at Gary invited prize submissions from teams with advanced ideas, early-stage ideas, and nascent concepts. Five advanced winners received $100,000 each, five early-stage winners received $80,000 each, and five beginning-concept winners receiving varying shares of $100,000. Gary also recognized seven teams, including one from a Colorado Springs-based network of child care centers, that didn’t win money but offered promising ideas.

The contest used an online platform that made each submission publicly viewable and allowed teams to get feedback from fellow candidates, and in some cases, mentoring from experts.

“We really wanted the prize to be an engaging opportunity for people in the early childhood field,” said Steffanie Clothier, Gary’s child development investment director.

Gary received 570 submissions, with winning ideas coming from nonprofit and for-profit groups, universities, city governments, and the National Head Start Association.

One winning team aims to eradicate book deserts by putting children’s reading materials in public spaces like barber shops and beauty salons. Another proposes classes on mindfulness to reduce child care providers’ stress levels. Several feature technology solutions — to improve child care business operations or promote early developmental screenings.

Clothier said although most of the prize winners are testing projects outside Colorado, their ideas could eventually be replicated here. She said the organization has not decided whether to hold the innovation competition again.