Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

Frequently asked

New Denver teacher contract: We answer the most common questions about the tentative pact

PHOTO: Joe Amon/The Denver Post
Students in class at Dora Moore ECE-8 during the second day of the Denver Public Schools teachers strike.

One reason many Denver educators didn’t like the district’s old ProComp pay system was that it was too complicated and unpredictable. Both sides agree that the deal reached early Thursday morning creates a much simpler pay system for teachers.

But educators — and the general public — still have a lot of questions about the tentative ProComp agreement, which still needs to be ratified by union members and the Denver school board. Here we’ve answered some of the most common questions we’ve heard since the end of the strike.

How do I place myself on the salary schedule?

The salary schedule is made up of “steps” and “lanes.” The “steps” represent years of service for which a teacher had a positive evaluation. The “lanes” represent levels of education. The new schedule has 20 steps and seven lanes.

Worked in Denver Public Schools for five years and have a master’s degree? Go to step five and then slide your finger over to the master’s degree lane. That’s your base salary.

Did you have a year when your evaluation wasn’t good? Go back one step. Have an additional 18 credits on top of your master’s degree? Go up one more lane.

Teachers can also go up a lane once they hit the 10-year mark because the district wanted to reward longevity. Other milestones that merit a lane change: earning national board certification or an advanced license, or completing six “professional development unit” training courses.

Still not sure? Denver Public Schools plans to put a salary calculator on its website soon.

What if I have more than 20 years of experience?

If you have 20 or more years of experience, you’re placed at the top of the salary schedule, on step 20. After step 20, you’ll get yearly cost-of-living raises. You’re still eligible to change lanes, but you won’t get any more step raises.

Does the district know everything it needs to know about individual educators to pay them the correct salary?

Denver Public Schools plans to send letters or emails this spring to every teacher and special service provider (nurses, counselors, and others) covered by the contract, laying out where the district believes that employee falls on the schedule based on information they have on file. Educators will have a certain amount of time to correct any wrong information and get on the correct step and lane for the 2019-20 school year.

Under the new salary schedule, it looks like I’ll earn less next year than I do now. Am I taking a pay cut?

No. The agreement includes a “hold harmless” clause that ensures everyone will get a raise next year. Those whose salaries are higher now than they would be under the new schedule will get a cost-of-living raise each year until the salary schedule catches up with them.

How are bonuses and incentives different under the new contract?

The bonuses and incentives are different in three ways: There are fewer of them, the dollar amounts are different, and the dollar amounts won’t change year to year.

This year, there are six bonuses and incentives offered by the district: one for educators who work in Title I schools where 60 percent or more of the student population qualifies for subsidized meals; one for educators who work in hard-to-fill positions; one for educators who work in “hard-to-serve” schools; one for educators who work in one of 30 “highest-priority” schools; one for educators who return year over year to those schools; and one for educators who work in schools deemed top-performing or high-growth, as based on school ratings.

Here’s what’s left in the new contract: Teachers in Title 1 schools and those in hard-to-fill positions, such as secondary math, will get $2,000 a year. Teachers who return year over year to 30 highest-priority schools will get $3,000 a year. Teachers in 10 schools deemed “distinguished” will get $750 a year, with the criteria to be determined by the district and the union.

Why aren’t the district and the union tying bonuses to test scores anymore?

Unions have traditionally been skeptical of paying teachers based on student test scores because the scores are so closely correlated with factors like race and household income. In Denver, these bonuses were also less predictable for teachers because the district often changed the criteria it used to rate schools and award “top-performing” bonuses.

The district also came to see these bonuses as canceling out the effects of bonuses for teachers at high-poverty schools. A teacher could get nearly the same kind of monetary reward by moving to a more affluent school or by staying in one where students face more challenges. The new bonus system provides clearer monetary benefits to working in a high-poverty school.

Why did the union agree to keep the incentive for highest-priority schools, when that had been such a sticking point?

In any negotiation, there’s give and take and a lot of moving pieces. 

Here’s what lead negotiator Rob Gould said to district officials during bargaining: “We are open to the incentive because we know it’s important to you. And we’re willing to entertain your ideas if we can get the base salary schedule that our teachers need. Because if we can get the base salaries we need, we can keep our teachers in Denver.”

This was also an issue that divided teachers, with some teachers at schools that received the highest-priority incentive pushing to keep them.

Did teachers get a better deal out of the strike than the district’s last offer before the strike?

Teachers were getting a raise no matter what. The district was offering an average 10 percent raise before the strike (this included a cost-of-living raise that was agreed to back in 2017). Now teachers will get an average 11.7 percent raise, though individual teachers will see a wide range.

The district is putting the same amount of new money — $23.5 million — into teacher compensation as it was offering before the strike. It can give a larger average raise with that same amount of money because the incentives are smaller than under the previous proposal and because of limits on how teachers can use training to get raises. That gives the district more predictability about how many teachers will get raises each year.

Union leaders call the deal a win. They secured more opportunities for teachers to earn raises and move into higher categories on the salary schedule, including through completing training partially during work hours at no additional cost. And teachers can get to $100,000 in 20 years, rather than the 30 years in the last district proposal.

However, individual teachers aren’t necessarily getting more base pay next year than they would have under the district’s last offer. Early-career teachers without advanced degrees would have earned more in base pay under the district’s last offer. The teachers who do better under the deal reached after the strike are veteran educators with more education.

To take two examples: A second-year educator with a bachelor’s degree and no extra credits or training would have earned $47,550 in base pay under the district’s last offer before the strike but will earn $46,869 under the deal reached this week.

But a 20-year educator who has a master’s degree and an advanced license who has been with the district for 10 years will earn $88,907 in base pay under the new agreement, compared with $87,550 under the district’s last proposal before the strike.

The union fought for this kind of salary schedule in part to address a longstanding complaint that teachers have little reason to stay in a district where base pay levels off.

You can see the salary schedule from the district’s last offer here and the schedule from the tentative agreement here.

Is this deal financially sustainable for the district?

Denver Public Schools Chief Financial Officer Mark Ferrandino says that is the “million-dollar question,” perhaps closer to the “half-billion-dollar question,” since that is roughly how much the district spends on educator compensation.

Ferrandino believes the answer is yes, with the standard caveat that all projections are just that.

What will be cut to pay for this?

The district plans to cut $20 million from administrative costs over the next two years. That includes cutting 150 jobs in the central office and ending all executive bonuses. The bulk of it — $13 million — will go to fund the ProComp agreement.

District officials have not yet said which central office jobs will be cut, though Superintendent Susana Cordova has said cuts will be to “discretionary” departments. Departments that will not be cut include special education, English language acquisition, and transportation, she said.

Teachers will get a raise. What about paraprofessionals, bus drivers, custodians, and cafeteria workers?

These other district employees, much lower paid than teachers, are not covered by the contract that was the subject of the strike. Cordova has said these workers also deserve raises and a portion of administrative cuts will go to pay for them.

But how much of a raise will they get? That will all be worked out over the next few months and include discussions with the unions that represent these employees.

Will striking teachers get back pay?

Not according to district officials. After this story was published Friday, we asked for further clarification on this. We received this statement Saturday morning:

Superintendent Cordova understands that when teachers make the choice to strike, they are doing so to make a statement and bring attention to the importance of the issue at hand. Foregoing pay during the time that a teacher is not working is a challenging decision that no one makes lightly, and consequently, brings with it an impact that is intended to push for change.

DPS did not feel that it would be fair or appropriate to provide back pay to striking teachers when many others — including more than 40 percent of classroom teachers — chose to remain at work this week. However, DPS is working with the DCTA to offer all teachers the opportunity to attend a Saturday session to replace the professional development day that was cancelled in the days leading up to the strike. Any teacher who attends will be paid a day’s salary.

When will the new agreement go into effect? How long will it last?

Assuming both sides ratify it, the new agreement technically (and retroactively) went into effect Jan. 19, the day after the old one expired. But educators won’t start receiving the new salaries, incentives, and bonuses negotiated under it until Aug. 1. The agreement expires Aug. 31, 2022.

Teens Talk Back

‘Mr. Mayor, we cannot afford to wait.’ Teen group says New York City diversity plan doesn’t move fast enough.

PHOTO: Courtesy/Teens Take Charge
Teens Take Charge members at a "virtual" press conference in New York City on Thursday

A teen group representing students from more than 30 New York City high schools sharply criticized a recent report from Mayor Bill de Blasio’s School Diversity Advisory Group as offering no real solutions for increasing integration in the city’s starkly segregated high schools.

At a virtual press conference on Thursday, broadcast live on Facebook by Teens Take Charge, students expressed support for the report’s broad policy aim of achieving greater integration but also disappointment that the findings offered few specifics for how to reach this goal. The mayor’s Diversity Advisory Group has said a follow-up report will provide more details later this year.

“We have been told to wait, to be patient, that change is coming soon,” said Tiffani Torres, a junior at Pace High School in Manhattan. “Mr. Mayor, we cannot afford to wait any longer.”

Teens Take Charge has long advocated for greater efforts to end segregated enrollment patterns in the city’s high schools. Sokhnadiarra Ndiaye, a junior at Brooklyn College Academy High School, said that students’ expectations of the mayor included his announcing “a comprehensive plan” — even if it took years to realize — “to racially, socioeconomically, and academically integrate high schools before the end of this school year,” she said.

Among Teens Take Charge’s specific recommendations are doing away with academic screens for admission to the city’s high schools, a more transparent process for applying to them, and more resources for low-income schools. Early last year, the group produced an Enrollment Equity Plan for increasing educational opportunities for low-income black and Hispanic students.

And because concrete plans for increasing integration would take time, Ndiaye said the teen organization supports several interim measures as well to address inequities in the school system. These include providing more college and career counseling for junior and seniors at low-income, under-resourced high schools. The teen group would also like to see the city provide vouchers to low-income families to access extra-curricular activities and programs offered by private companies or the ability to participate in such programs at other public schools if theirs don’t offer them. (Some city teens joined a class-action lawsuit against the education department and Public School Athletic League for allegedly denying black and Hispanic students equal opportunity to play on school sports teams, in violation of local human rights law.)

Torres described how Teens Take Charge has had “several meetings and phone conversations with Department of Education officials over the past year,” and schools chancellor Richard Carranza has stated that students have his ear. “We’re listening,” he tweeted in response to a Chalkbeat story with excerpts of the students’ views.

In December, the city’s education department posted a new job listing for a “Student Voice Manager” who would gather students’ thoughts on education policies. But while acknowledging this seat at the table, several students expressed frustration at the slow pace of change.

Bill de Blasio’s office declined to comment about Teens Take Charge’s concerns or their specific recommendations, beyond referencing remarks the mayor already made about the School Diversity Advisory Group report.

Doug Cohen, an education department spokesman, said in a statement, “We’ve taken real steps toward school integration,” pointing to initiatives such as a $2 million diversity grant program for school districts and communities citywide to develop their own local diversity plans, and a program that enables middle-schoolers to visit college campuses. “We know there is more work to do, and we thank Teens Take Charge for its continued advocacy on these issues,” he added.

Students at the group’s event urged swift change. “They know our plan; they have our information,” said Sophie Mode, a sophomore at Brooklyn Millennium. “They need to take action now.”