data-driven decisionmaking

Why we won't publish individual teachers' value-added scores

Tomorrow’s planned release of 12,000 New York City teacher ratings raises questions for the courts, parents, principals, bureaucrats, teachers — and one other party: news organizations. The journalists who requested the release of the data in the first place now must decide what to do with it all.

At GothamSchools, we joined other reporters in requesting to see the Teacher Data Reports back in 2010. But you will not see the database here, tomorrow or ever, as long as it is attached to individual teachers’ names.

The fact is that we feel a strong responsibility to report on the quality of the work the 80,000 New York City public school teachers do every day. This is a core part of our job and our mission.

But before we publish any piece of information, we always have to ask a question. Does the information we have do a fair job of describing the subject we want to write about? If it doesn’t, is there any additional information — context, anecdotes, quantitative data — that we can provide to paint a fuller picture?

In the case of the Teacher Data Reports, “value-added” assessments of teachers’ effectiveness that were produced in 2009 and 2010 for reading and math teachers in grades 3 to 8, the answer to both those questions was no.

We determined that the data were flawed, that the public might easily be misled by the ratings, and that no amount of context could justify attaching teachers’ names to the statistics. When the city released the reports, we decided, we would write about them, and maybe even release Excel files with names wiped out. But we would not enable our readers to generate lists of the city’s “best” and “worst” teachers or to search for individual teachers at all.

It’s true that the ratings the city is releasing might turn out to be powerful measures of a teacher’s success at helping students learn. The problem lies in that word: might.

Value-added measures do, by many readings, appear to do the job that no measure of a teacher’s quality has done before: They estimate the amount of learning by students for which a teacher, and no one else, is responsible, and they do this with impressive reliability. That is, a teacher judged to be more effective one year by value-added is likely to continue to be judged effective the next year, and the year after that.

But this is not true for every teacher — hardly. Many teachers will be mislabeled; no one disputes this. Value-added scores may be more reliable than existing alternatives, but they are still far from perfectly reliable. It’s completely possible, for instance, that a teacher judged as less effective one year will be judged as very effective the next, and vice versa.

As we reported two years ago, when the NYU economist Sean Corcoran looked at New York City’s value-added data, he found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

The fluctuation is acknowledged by even the strongest supporters of using value-added measures to evaluate teachers. One of the creators of the city’s original value-added model, the Columbia economist Jonah Rockoff, compares value-added scores to baseball players’ batting averages. One of his reasons: In each case, the year-to-year fluctuations of an individual’s score are about the same.

“If someone hit, you know, .280 last year, that doesn’t guarantee they’re going to hit .280 next year,” Rockoff said today. “However, if you hit .210 last year and I hit .300, there’s a very high likelhood I’m going to hit more than you next year, too. Whereas if you hit .280 and I hit .278, we’re basically the same.”

Another challenge is that many researchers still aren’t convinced that value-added scores are measuring the right sort of teacher impact. The challenge lies in the flaws of the measures on which value-added scores depend — standardized state test scores.

Tests are supposed to measure what a student has learned about a subject, but they can also reflect other things, like how well her teacher prepared her for the test, or how well she mastered the narrow band of the subject the test assessed.

The test-prep concern is magnified by findings that a single teacher can generate two different value-added scores if evaluators use two different student tests to determine them. The Gates Foundation’s Measures of Effective Teaching study calculated value-added scores for teachers based on both state tests and more conceptual tests. They found substantial differences between the two, according to an analysis by the economist Jesse Rothstein of the University of California at Berkeley.

“If it’s right that some teachers are good at raising the state test scores and other teachers are good at raising other test scores, then we have to decide which tests we care about,” Rothstein said today. “If we’re not sure that this is the test that captures what good teaching is, then we might be getting our estimates of teaching quality very wrong.”

Flags about exactly what high value-added ratings reward are also raised by studies that ask how the ratings match up with measures of what teachers actually say and do in the classroom. Heather Hill,  professor at Harvard’s Graduate School of Education, rated math teachers’ teaching quality based on an observation rubric called the Mathematical Quality of Instruction, which looks at factors like whether the teacher made mathematical errors and the quality of her explanations. Then Hill compared the math teaching rating to value-added measures.

Two individual cases stood out: One teacher had made a slew of math errors in her teaching, and the other had failed to connect a class activity to math concepts. But teachers’ value-added scores put them at the top of their cohort.

There is some reason to think that value-added measures reflect more than test prep. Rockoff points out that while different tests can produce different value-added scores for the same teacher, the two measures are still correlated. Using different tests, he said, is akin to looking at slugging percentage rather than batting average. “I’m sure those two things are positively correlated, but probably not one for one,” he said.

More persuasively, a recent study by Rockoff and two other colleagues concluded that value-added measures can actually predict long-term life success outcomes, including higher cumulative lifelong income, reduced chance of teen pregnancy, and living in a high-quality neighborhood as an adult. The study examined an anonymous very large urban school district that bears several similarities to New York City.

That study targeted another concern about value-added measures: that teachers score consistently well year after year not because of something they are doing, but because they consistently teach students with certain advantages.

Rothstein has used value-added models to conclude that fifth-grade teachers have strong effects on their students’ performances in third-grade — something they could not possibly influence, unless value-added scores reflect not just teachers’ influence but also advantages brought by students.

Rockoff and his colleagues evaluated the possibility by testing a question. If high-value added teachers do well because they get the “better” students of those in their grade, then their students’ high test score growth would be linked with mediocre performance in other classrooms. That would mean that, when researchers looked at growth for the entire grade, the “better” students’ growth would be canceled out by their less lucky peers. But the scores were not canceled out, suggesting that effective teachers did more than just have unusually good students.

None of this means that we won’t write about what the data dump includes or that we might not publish an adapted database that strips out information linking the city’s data to individual teachers. With more than 90 columns in the Excel sheet the city has developed — and more than 17,000 rows, representing the number of reports issued over their two-year lifespan — the release might well enable us to examine the city’s value-added experiment in new ways.

Value-added measures certainly aren’t going away. City officials only stopped producing Teacher Data Reports because they knew the State Education Department is preparing its own. The measures, which are expected to come out in 2013, will make up 25% of the evaluation for teachers of math and English in tested grades.

college plans

As Washington decides their fate, ‘Dreamers’ preparing for college are stuck in limbo

PHOTO: Joe Amon/The Denver Post
Randi Smith, a psychology teacher at Metro State University, marched to support Deferred Action for Childhood Arrivals during a citywide walkout in downtown Denver, CO.

While many high schoolers spend spring of their senior year coasting through classes and waiting to hear back from colleges, undocumented students who hope to attend college spend their time calling lawyers, consulting school counselors, and scouring the internet in search of ways to pay for school without the help of federal financial aid or student loans — assuming they even get in.

That process, anxiety-provoking even in a normal year, has become incalculably more chaotic this admissions season — even traumatic — as these young undocumented immigrants watch President Trump and lawmakers wrangle over Deferred Action for Childhood Arrivals, the program that has until now allowed them to remain in the country without having to fear deportation.

As the policy battle nears a climax, these students aren’t just breathlessly waiting to learn whether they’ll be accepted into college — they’re waiting to see whether they have a future in this country.

“It’s different for me. It’s definitely more stressful and there are times when you want to give up,” said an undocumented student at KIPP NYC College Prep High School, who is graduating this year and applying to colleges. She requested anonymity because of her legal status. “But then I remind myself that regardless of what’s going on, I’m still going to do what I’ve set myself to do.”

High school counselors are also feeling the strain. They already faced the difficult task of helping undocumented students compete for private scholarships, and finding schools that will support those students once they’re on campus. Now those counselors also must monitor each twist and turn of the immigration debate in Washington, while, somehow, trying to keep their undocumented students focused on college.

One of those counselors is John Kearney, who works at Guadalupe Centers Alta Vista High School, a charter school in Kansas City, Missouri. Dozens of his soon-to-graduate students are beneficiaries of DACA, a program created under former President Obama that allows undocumented immigrants who were brought to the country as children to avoid deportation and work here legally. Lately, they have been asking him why they should even consider college when their fate in the U.S. is so uncertain.

“The big question is, ‘Why? Why go to college, and then I can’t even work, then why?’” said Kearney, who also helped start a nonprofit that provides scholarships to undocumented students. “It’s a really tough question.”

As of Friday, President Trump and lawmakers were still locked in heated negotiations over DACA, which Trump said this fall that he would eliminate unless Congress enshrined it in law. Without an agreement, it is set to expire March 5, just as graduating seniors firm up their college plans. If that happens, young immigrants, often called Dreamers, could lose the few crucial protections they have. For many, their DACA status has already lapsed.

Even with DACA’s protections, Dreamers face massive hurdles to enroll in college: They don’t qualify for federal aid or loans, and, in some states, are barred from receiving financial aid or even attending public universities. Out of the estimated 65,000 undocumented students who graduate from high school every year, only 5-10 percent enroll in college.

Following Trump’s announcement in September, counselors have also had to race against the clock counting down to DACA’s expiration: That meant juggling college application deadlines with the October cutoff for students to apply for renewed DACA status.

The KIPP charter school network received a donation this year to help students pay for the renewal fee, which has been a godsend for many students — including the young woman who is graduating from KIPP NYC College Prep High School.

As soon as she learned the school would pay the fee for her, she immediately called her father, who is also undocumented and repairs beauty-salon equipment for a living.

“My dad was definitely trying to round up the money before the deadline, so it was a blessing that the school was able to find a donor,” she said. “I told him not to worry about it and it was a relief — like a weight off his shoulders.”

If the girl was trying to relieve her father’s stress, her college counselor, Rob Santos, was trying to do the same for her. Even as she balanced college-application essays, transcripts, and the rest, she was also coming to realize how quickly her life would change if DACA is not extended.

“There was definitely extra emotional support that I’ve had to provide this year,” Santos said. “I definitely had my DACA student in my office, and tears were happening.”

Santos keeps a running list of the colleges that accept students who don’t have permanent legal status and the few scholarships available to them. Many of those scholarships require undocumented students to have DACA status. If the program ends, it’s unclear whether students will still be eligible.

Still, Santos said his dreamer student rarely talks about the political furor surrounding her future in the U.S. as she awaits her college-acceptance letter. Instead, she’s more likely to discuss her hope of one day studying business and fashion.

“Our DACA students are resilient. They’re optimistic,” Santos said. “But they’re also realistic for what could actually happen.”

blast from the past

Harkening back to earlier era, struggling New York City school fights closure but faces long odds

PHOTO: Christina Veiga
Kevin Morgan, the Parent Association president at P.S./M.S. 42, is leading a fight to keep the Rockaway school open.

A decade ago, teachers picketed P.S./M.S. 42 R. Vernam in Rockaway, Queens and declared the campus unsafe. Parents said the building was in horrible shape — some areas reeked of urine — and they petitioned the education department to close the school and start over.

But when Mayor Bill de Blasio took office, he had a different idea: Rather than shut its doors, he would revamp it. After three years in de Blasio’s “Renewal” improvement program, which injects troubled schools with academic supports and social services, P.S./M.S. 42 appeared to be making progress: Its test scores and quality reviews have steadily improved. Enrollment, while lower this year, has mostly been stable.

So when the education department announced plans last month to shutter P.S./M.S. 42 and 13 other low-performing schools, many in the school community were shocked.

“We think that this is a mistake,” said Donovan Richards, the local city councilman who said that when he met with schools Chancellor Carmen Fariña shortly before the announcement, far from declaring the school a lost cause, she praised its recent strides and discussed ways to celebrate them.

“You have this glimmer of hope and turnaround in the building,” he added, “and yet we’re reversing the progress.”

PHOTO: Christina Veiga
Parents used to complain about poor conditions at P.S./M.S. 42, which has since built a new addition.

Now, parents, teachers and local political leaders are vowing to fight its closure. The coalition has launched an aggressive social media campaign, printed highlighter-yellow T-shirts declaring the school “strong and united,” and planned rallies at the school and in Albany, where the school’s supporters traveled Tuesday to make their case to state lawmakers.

On a recent morning, Kevin Morgan, the school’s parent association president, went to his local congress member’s office to appeal for help, and brought in a motivational speaker to inspire students as they drafted essays in defense of their school.

“It’s not fair,” he said. “They need to rethink what they’re about to do. How is this going to affect these children?”

The fight puts the mayor in the uncomfortable position of defending the closure of a low-performing school despite signs of improvement and vocal opposition from some parents — a scenario he railed against when running to replace then-Mayor Michael Bloomberg. At the time, de Blasio blasted Bloomberg for disregarding the will of parents in his zeal to shutter and replace troubled schools without first giving them a chance to rebound.

Now, after investing $582 million in a program meant to offer bottom-ranked schools the second chance he said they had been denied, de Blasio finds himself coming to the same conclusion as his predecessor: Some underachieving schools simply can’t be resuscitated — at least not quickly enough — so better to pull the plug and start fresh.

“After a serious effort, we do not think, with their current structures, they can make it,” de Blasio said on NY1 the day the closures were announced. Still, he defended the turnaround effort, saying that, without it, “we would have continued to see closures without an honest effort to fix the problem.”

In the case of P.S./M.S. 42, the education department is proposing to replace it with two new schools — an elementary school and a middle school — in the same building.

It’s likely they will serve many of the same students as the school they’re supplanting, though some parents worry the new schools may deploy admissions criteria that will screen out some of P.S./M.S. 42’s current students. An education department spokesman said the new schools would not turn away any P.S./M.S. 42 students. The new schools may also employ many of the same teachers, under a contract rule that says at least half the positions in replacement schools must be offered to teachers at closed schools who apply and hold the right qualifications.

P.S./M.S. 42 boosters hope the new schools never have a chance to open. But they face long odds: Under de Blasio, very few schools on the chopping block have managed to escape.

Last year, the Panel for Educational Policy — an oversight board where the majority of members are appointed by the mayor — signed off on all of the city’s proposed closures. Even when parents at J.H.S 145 in the Bronx mounted a campaign to keep the middle school open, only five of the 13 panel members voted against its closure.

The city’s plan to shutter P.S./M.S. 42 follows a yearslong, grassroots effort to save it.

Today, one of the leaders of that campaign is an unlikely champion: a parent named Queen Makkada, who called for the school’s closure in 2010 when her two children went there. At one point, her daughter was attacked by a group of boys, and students were known to roam the hallways unsupervised.

PHOTO: Christina Veiga
Queen Makkada says P.S./M.S. 42 has struggled in the past, but is now showing improvement.

“We literally had first graders cutting class,” Makkada said. A joint city-state report from 2011 said teachers there “demanded little” from students and parents complained about unchecked bullying among students.
Makkada says things began to turn around when the current principal, Patricia Finn, took over about seven years ago. Finn did not respond to a request for comment.

The principal smoothed over relations with teachers, who have filed far fewer grievances under her than the previous administration, according to their union. And she forged relationships with skeptical parents, Makkada said. Last year, 90 percent of parents who responded to a school survey said the principal works to build community.

“All the stakeholders had to come together and change it,” Makkada said. “These parents went through the process to improve a failing school.”

At the same time that parents were getting more involved, the school facilities were getting an upgrade. In 2011, a gleaming new addition was built onto the building, and there are plans for a new $7 million playground, according to the city councilman.

The Renewal program, which launched in 2014, marked a new wave of investment in P.S./M.S 42. A community-based nonprofit — Family Health International, which goes by FHI 360 — brought much-needed mental health supports for students, including one-on-one counseling. The school day was extended by an hour. And the school has launched several initiatives aimed at improving school culture, including training students to help resolve conflicts among their peers, parents said.

Since 2014, the school has received improved “quality review” ratings from official observers, and its test scores have ticked upwards. In fact, elementary students at P.S./M.S. 42 earned higher scores on the state English and math tests last year than the average among Renewal schools that the city is keeping open. Its middle-school students perform just below that average.

And enrollment, a key factor that chancellor Fariña says the education department considers when recommending closures, grew by dozens of students the first few years of the program. This past year, its population declined to just over 660 students — but that’s still higher than before becoming a Renewal school.

Given the progress, parents don’t understand why their school is targeted for closure.

“This is ripping everything apart,” said Morgan, the parent-association president.

But despite the recent improvements, the majority of the school’s students still are far behind where they should be.

Only 17 percent of elementary students and 14 percent of middle schoolers passed last year’s state English tests — compared with 40-41 percent of students citywide. In math, 14 percent of elementary students and 6 percent of middle schoolers passed the tests, compared with 42 percent and 33 percent citywide.

Meanwhile, a stubbornly high share of students are chronically absent, despite a major push by the city to boost attendance at Renewal schools. More than 45 percent of P.S./M.S. 42 students miss 10 percent or more of the school year, compared to 36 percent among all Renewal schools and about 26 percent among all city schools, according to the education department.

“This decision to propose a school closure was made based on a careful assessment of the school community as a whole,” Aciman, the department spokesman, said in a statement. He added that community engagement is an important part of proposed closures, and said officials will respond to parents’ questions and concerns.

Officials will hold a public hearing at the school on Jan. 10, before the Panel for Educational Policy votes Feb. 28 on whether to approve the city’s closure plans.

Among the P.S./M.S. 42 parents who will ask the panel to spare the school is Willard Price.

He said teachers have given his son, William, extra help in math and handwriting, and principal Finn has invited him to eat lunch in her office when he felt overwhelmed by the cafeteria. Now, William earns high marks on his report cards and would like to remain at P.S./M.S. 42 for middle school, his father said.

“I think that’s messed up, trying to close the school,” William said. “This school is the only school I ever liked.”