data-driven decisionmaking

Why we won't publish individual teachers' value-added scores

Tomorrow’s planned release of 12,000 New York City teacher ratings raises questions for the courts, parents, principals, bureaucrats, teachers — and one other party: news organizations. The journalists who requested the release of the data in the first place now must decide what to do with it all.

At GothamSchools, we joined other reporters in requesting to see the Teacher Data Reports back in 2010. But you will not see the database here, tomorrow or ever, as long as it is attached to individual teachers’ names.

The fact is that we feel a strong responsibility to report on the quality of the work the 80,000 New York City public school teachers do every day. This is a core part of our job and our mission.

But before we publish any piece of information, we always have to ask a question. Does the information we have do a fair job of describing the subject we want to write about? If it doesn’t, is there any additional information — context, anecdotes, quantitative data — that we can provide to paint a fuller picture?

In the case of the Teacher Data Reports, “value-added” assessments of teachers’ effectiveness that were produced in 2009 and 2010 for reading and math teachers in grades 3 to 8, the answer to both those questions was no.

We determined that the data were flawed, that the public might easily be misled by the ratings, and that no amount of context could justify attaching teachers’ names to the statistics. When the city released the reports, we decided, we would write about them, and maybe even release Excel files with names wiped out. But we would not enable our readers to generate lists of the city’s “best” and “worst” teachers or to search for individual teachers at all.

It’s true that the ratings the city is releasing might turn out to be powerful measures of a teacher’s success at helping students learn. The problem lies in that word: might.

Value-added measures do, by many readings, appear to do the job that no measure of a teacher’s quality has done before: They estimate the amount of learning by students for which a teacher, and no one else, is responsible, and they do this with impressive reliability. That is, a teacher judged to be more effective one year by value-added is likely to continue to be judged effective the next year, and the year after that.

But this is not true for every teacher — hardly. Many teachers will be mislabeled; no one disputes this. Value-added scores may be more reliable than existing alternatives, but they are still far from perfectly reliable. It’s completely possible, for instance, that a teacher judged as less effective one year will be judged as very effective the next, and vice versa.

As we reported two years ago, when the NYU economist Sean Corcoran looked at New York City’s value-added data, he found that 31 percent of English teachers who ranked in the bottom quintile of teachers in 2007 had jumped to one of the top two quintile by 2008. About 23 percent of math teachers made the same jump.

The fluctuation is acknowledged by even the strongest supporters of using value-added measures to evaluate teachers. One of the creators of the city’s original value-added model, the Columbia economist Jonah Rockoff, compares value-added scores to baseball players’ batting averages. One of his reasons: In each case, the year-to-year fluctuations of an individual’s score are about the same.

“If someone hit, you know, .280 last year, that doesn’t guarantee they’re going to hit .280 next year,” Rockoff said today. “However, if you hit .210 last year and I hit .300, there’s a very high likelhood I’m going to hit more than you next year, too. Whereas if you hit .280 and I hit .278, we’re basically the same.”

Another challenge is that many researchers still aren’t convinced that value-added scores are measuring the right sort of teacher impact. The challenge lies in the flaws of the measures on which value-added scores depend — standardized state test scores.

Tests are supposed to measure what a student has learned about a subject, but they can also reflect other things, like how well her teacher prepared her for the test, or how well she mastered the narrow band of the subject the test assessed.

The test-prep concern is magnified by findings that a single teacher can generate two different value-added scores if evaluators use two different student tests to determine them. The Gates Foundation’s Measures of Effective Teaching study calculated value-added scores for teachers based on both state tests and more conceptual tests. They found substantial differences between the two, according to an analysis by the economist Jesse Rothstein of the University of California at Berkeley.

“If it’s right that some teachers are good at raising the state test scores and other teachers are good at raising other test scores, then we have to decide which tests we care about,” Rothstein said today. “If we’re not sure that this is the test that captures what good teaching is, then we might be getting our estimates of teaching quality very wrong.”

Flags about exactly what high value-added ratings reward are also raised by studies that ask how the ratings match up with measures of what teachers actually say and do in the classroom. Heather Hill,  professor at Harvard’s Graduate School of Education, rated math teachers’ teaching quality based on an observation rubric called the Mathematical Quality of Instruction, which looks at factors like whether the teacher made mathematical errors and the quality of her explanations. Then Hill compared the math teaching rating to value-added measures.

Two individual cases stood out: One teacher had made a slew of math errors in her teaching, and the other had failed to connect a class activity to math concepts. But teachers’ value-added scores put them at the top of their cohort.

There is some reason to think that value-added measures reflect more than test prep. Rockoff points out that while different tests can produce different value-added scores for the same teacher, the two measures are still correlated. Using different tests, he said, is akin to looking at slugging percentage rather than batting average. “I’m sure those two things are positively correlated, but probably not one for one,” he said.

More persuasively, a recent study by Rockoff and two other colleagues concluded that value-added measures can actually predict long-term life success outcomes, including higher cumulative lifelong income, reduced chance of teen pregnancy, and living in a high-quality neighborhood as an adult. The study examined an anonymous very large urban school district that bears several similarities to New York City.

That study targeted another concern about value-added measures: that teachers score consistently well year after year not because of something they are doing, but because they consistently teach students with certain advantages.

Rothstein has used value-added models to conclude that fifth-grade teachers have strong effects on their students’ performances in third-grade — something they could not possibly influence, unless value-added scores reflect not just teachers’ influence but also advantages brought by students.

Rockoff and his colleagues evaluated the possibility by testing a question. If high-value added teachers do well because they get the “better” students of those in their grade, then their students’ high test score growth would be linked with mediocre performance in other classrooms. That would mean that, when researchers looked at growth for the entire grade, the “better” students’ growth would be canceled out by their less lucky peers. But the scores were not canceled out, suggesting that effective teachers did more than just have unusually good students.

None of this means that we won’t write about what the data dump includes or that we might not publish an adapted database that strips out information linking the city’s data to individual teachers. With more than 90 columns in the Excel sheet the city has developed — and more than 17,000 rows, representing the number of reports issued over their two-year lifespan — the release might well enable us to examine the city’s value-added experiment in new ways.

Value-added measures certainly aren’t going away. City officials only stopped producing Teacher Data Reports because they knew the State Education Department is preparing its own. The measures, which are expected to come out in 2013, will make up 25% of the evaluation for teachers of math and English in tested grades.

Superintendent search

Former principal Roger Leon chosen as Newark’s new superintendent

Former principal and veteran administrator Roger Leon has been chosen as Newark’s new schools chief — its first since the city regained control of its schools.

In a unanimous vote Tuesday night, the school board chose Leon — a Newark native backed by local elected officials — over two candidates with extensive experience in other large urban districts, but whose outsider status put them at a disadvantage. The son of Cuban immigrants, Leon takes the reins of a system whose population has become increasingly Hispanic: At 46 percent of the Newark Public Schools enrollment, Hispanic students now outnumber black students, who make up 44 percent of the enrollment.

In opting for Leon, the board also passed over A. Robert Gregory, another former Newark principal and the district’s interim superintendent, who rose through the ranks under the previous state-appointed superintendent, Christopher Cerf — which some critics saw as a blemish on his record. The board actually picked Leon as superintendent once before, in 2015. But the state education commissioner, who still controlled the district at that time, ignored the board’s choice and appointed Cerf.

The board’s decision to again tap Leon seemed to signal a definitive break from the era of sweeping, controversial changes enacted by outsiders — namely, Cerf and his predecessor, Cami Anderson. Instead, after the state ended its decades-long takeover of the district in February and put the board back in charge of the schools, the board’s choice for superintendent suggests that it will rely on local talent and ideas to guide New Jersey’s largest school system in the new era of local control.

“After 22 years of being under state control, this is a new day,” said School Board Chair Josephine Garcia after Tuesday’s vote. “We look forward to working with the new superintendent.”

Leon grew up in Newark’s Ironbound neighborhood, where he attended the Hawkins Street School. He graduated from Science Park High School, the highly competitive magnet school, where he returned as a substitute math teacher while still a student at Rutgers University. He later coached the school’s renowned debate team.

He went on to teach middle-school algebra, then became principal of Dr. William H. Horton School and later University High School of the Humanities. For the past decade, he has been an assistant superintendent in the district.

As deputy chief academic officer under former superintendent Clifford Janey, he helped oversee several major policy changes, including new graduation requirements and district-wide grading standards. During that process, he recruited hundreds of parents, experts, and community members to join advisory committees to help craft the new policies.

More recently, he has played less of a policymaking role, instead helping to organize district-wide initiatives like a book-giveaway program for students. He also often authors the proclamations that the district awards to distinguished students and educators.

At a forum on Friday where the four superintendent finalists introduced themselves to the public, Leon said the district needs “a clear direction” for the future. He said his vision includes an “advanced technological curriculum” in schools, a focus on social-emotional learning, teacher training, and public-private partnerships to bring additional resources into schools.

“I will inherently be a proficient and influential agent of change,” he said, “because anything short of that is unacceptable.”

Leon arrives in his new position with a strong base of support, which was evident after Tuesday’s vote, when the audience erupted into cheers. In addition to the many parents and educators he has crossed paths with during his 25 years working in the district, he is also said to have close ties with State Sen. Teresa Ruiz, an influential lawmaker based in the politically powerful North Ward.

While Leon served under both Anderson and Cerf, he was far enough removed from the decision-making to escape the wrath of critics who opposed their policies, which included closing some district schools and overseeing the expansion of the charter-school sector. On Tuesday, John Abeigon, the head of the Newark Teachers Union, which clashed bitterly with Anderson and Cerf, said he looked forward to working with Leon.

“Once the new superintendent is sworn in,” he said, “we can begin rebuilding some of the more positive aspects of our district that were destroyed under the corporate control of Cerf.”

While the board has now officially offered Leon the position, it must still negotiate the terms of his contract. He will then start his new role on July 1.

Leon was one of four finalists selected by a search committee after a national search. A state plan had called for the board to choose from just three finalists. But someone on the search committee was unhappy with the three who were chosen and asked the state commissioner to allow a fourth finalist — despite the objections of some other committee members.

While the audience at Tuesday’s board meeting loudly cheered the board’s final decision, many people still criticized the search process. The board kept the names of the finalists secret until shortly before Friday’s forum, where audience members were not permitted to ask the candidates questions.

Still, even critics of the process said they were eager to work with the superintendent.

“The board made their decision,” said Wilhelmina Holder, a longtime parent activist. “So now we’re going to have to respect that decision and work on behalf of the children.”

Superintendent search

On eve of historic vote in Newark, questions arise about superintendent selection process

PHOTO: Newark Press Information Office

When the Newark school board votes on a new superintendent Tuesday evening, as is expected, it will choose from four finalists — a notable departure from the state’s guidelines for the search, which called for a maximum of three finalists.

The change, the result of a behind-the-scenes dispute, is likely to raise questions about the integrity of the superintendent search process at a critical juncture, as the local school board takes control for the first time in over two decades.

The fourth finalist was added after a search committee had already agreed on its shortlist, and despite the objections of some committee members who wanted to stick with the initial three finalists, according to Kim Gaddy, a committee member and school board member, and Marques-Aquil Lewis, the former school board chair, who were both involved in the process.

The addition came at the insistence of other search committee members who were upset that a “strong” candidate had been left off the shortlist, according to Lewis. The additional name was added after the state education commissioner, who is overseeing the handover to local control, agreed to revise the state-authored playbook governing the transition.  

The identities of the four finalist candidates are public, but search committee members would not confirm which of the four was added to the list late.

The dispute over the superintendent selection process comes as the elected school board is choosing a schools chief for the first time since 1995, when the state seized control of the district. In February, the state provisionally returned control of the district to board, whose first major task is to choose a new superintendent.

Gaddy, the school board member who was on the seven-person search committee, said she did not even learn about the request for a fourth candidate until after it was sent. (Lewis, the board chairman who sent the request, disputes that.) Either way, Gaddy says the committee should have honored the process as it was written in the guidelines, which the district must adhere to in order to maintain control of its schools.

“When we finished with three members, that’s it. There should not have been any other discussion with the search committee,” said Gaddy, who declined to say who was the fourth finalist added to the list.

In order to fully return to local control, the district must follow a two-year state plan that spells out every detail of the transition. The plan stipulated that the board must conduct a national search for superintendent candidates, who would then be narrowed down to three finalists by the search committee.

During their deliberations, the committee members discussed the possibility of naming four finalists, but there was “no real consensus” on whether to ask for an additional finalist, according to Gaddy. So at its final meeting on April 21, the group decided to adhere to the plan and name three finalists.

However, immediately after that meeting, one or more members approached Lewis, who was then the chair of the school board, and asked him to send a request to the state asking for permission to name a fourth finalist, Lewis said. Lewis, who was not on the search committee, would not say who asked him to request the change. But he said they were unhappy with the shortlist of finalists.

“When the request was made, they felt there was a fourth candidate that was strong, that should have made the finals,” he said, adding that the person or persons did not tell him who the candidates were.

Lewis said he reached out to all seven committee members before making the request, but could not reach one member. (Lewis said he did speak with Gaddy, which she says she does not recall.)

Two members objected to the request, Lewis said. But he said that four agreed to it, so he sent a letter to the commissioner asking for a change to the transition plan.

Just after Lewis sent the request, he was replaced as board chair by Josephine Garcia. (Lewis did not run for re-election.) After becoming chair, Garcia re-sent the request to the state.

Once again, Gaddy said she was not informed in advance: “I found out after the fact. I was not asked to support it.” Instead, she said that Garcia said she would discuss the request at a board meeting — after it had already been sent. (Garcia did not respond to an email seeking comment.)

On April 27, Acting State Education Commissioner Lamont Repollet sent Garcia a letter saying her request had been granted.

“I am in receipt of your request to amend the Transition Plan to allow the Superintendent Search Committee to submit four finalists to the full Board of Education for consideration,” Lamont wrote in the letter, which the state education department provided to Chalkbeat.

“In order to provider greater assistance to the district in finding the best candidate for the Superintendent position and to allow for consideration of all potentially qualified candidates,” Lamont continued, he agreed to amend the transition plan to allow for four finalists.

After the request was granted, four finalists were presented to the school board — including the one who did not make the original list of three. The four introduced themselves to the public on Friday, and were interviewed by the board in private on Saturday. The full board is expected to vote on which finalist to extend the offer to at its meeting Tuesday evening.

The finalists are former Baltimore city schools chief Andres Alonso; Newark Interim Superintendent A. Robert Gregory; Newark Assistant Superintendent Roger Leon; and Sito Narcisse, chief of schools for Metro Nashville Public Schools in Tennessee.

The search committee includes three board members: Gaddy, Garcia, and Leah Owens. Three other members were jointly chosen by the mayor and the state education commissioner: Former Newark superintendent Marion Bolden, Rutgers University-Newark Chancellor Nancy Cantor, and Irene Cooper-Basch, executive officer of the Victoria Foundation. A seventh person, attorney Jennifer Carrillo-Perez, was appointed by the commissioner.

Only Gaddy would agree to speak on the record for this story; the other committee members did not respond to messages or declined to comment on the record.

Gaddy said she kept the names of the candidates confidential throughout the process, as required. However, she said she felt the entire process has been tainted by the decision to change the rules of the search without the agreement of the full search committee.

The transition plan “was a roadmap,” Gaddy said, that provided clear instructions: “‘You have two years to do A, B, C, and D.’”

“Now every time you don’t agree with A or you don’t agree with B, you’re going to write a letter to the commissioner?” she asked. “How is that following the plan and inspiring confidence in the ability to run this district?”