The science of setting ISTEP’s passing scores relies on a lot of judgment calls

The companies that make standardized tests like Indiana’s ISTEP exam would probably prefer not to call the process of setting the passing score guesswork.

But the first step of determining the cut-off score to pass ISTEP literally begins with a panel of educators making an educated and carefully considered guess. The process is widely accepted as valid, but it still falls well short of exact or scientific.

“It’s a political game, there’s nothing scientific about it,” said Ginette Delandshere, a test researcher from Indiana University-Bloomington. “But there’s no other way to do it either. If you have tests and you want to make decisions on the basis of those tests, you have to have cut scores. There’s no other way around it.”

Even after input from teachers and others is compiled and leads to the passing score test makers recommend, the process doesn’t end. It’s not over until the Indiana State Board of Education either agrees or changes the cut-off score to reflect its own thinking about how many questions students need get right to pass.

That’s what’s coming when the board meets next week.

Standardized tests are very different from the sorts of tests teachers give in schools because of all the have to accomplish. A test like ISTEP has to cover a broad range of skills and give information to show which students are well above or far below what is expected. That means some questions are intentionally very easy or very hard.

In a typical classroom, it’s easy to answer a fairly straightforward question of how much of the material a student learned. For example, if a student gets 48 right answers on a simple 50-question multiple choice test in math, it’s a 96 percent — an A.

But that won’t work on a standardized test.

The test also must be statistically “reliable,” or predictable in the sense that the passing rate should not to shift wildly from one year to the next. Questions also have to be written so they don’t reflect cultural or other biases.

To do all those things — include hard and easy questions, ensure questions are not biased, produce a score similar to last year and more — makes standardized exams more complex than other tests kids take in class.

On a standardized exam, the passing score might be sensibly set at 40 questions right out of 60, for example, even though that also means students only need to get 66 percent right, which would fail in any Indiana high school that sets 70 percent as a minimum.

Educators guide the process

The first step in setting a passing score for an ISTEP test begins with panels of educators.

In Indiana, the state board’s test director, Cynthia Roach, said they try to keep all passing score decisions based on what kids are expected to know according to the state’s academic standards.

Teachers and other educators are grouped by subject and grade level and are asked to create descriptions of what the “ideal” student would look like at each passing level.

They discuss questions like: What should a student who scores advanced, or “pass-plus,” know and be able to do in third-grade math? What kind of skills do seventh-grade reading students have to have to pass the exam? What specific skills make a student who passes student different from one who doesn’t? What skill level keeps a student on-track for college?

“It does come down to teachers, especially teachers who’ve taught for awhile because they’ve seen which kids go on and which don’t,” Roach said. “They get a feel, and that’s why they’re called the content experts.”

With the questions in mind as a mental guide, the panels flip through booklets with all of the test questions ordered from easy to difficult. They’re asked to decide where they think the test gets too hard for the average student.

The panels discusses their choices for how many questions they think it should take to pass and reconsider until they arrive at final recommendations for passing scores that must go to the state board.

Politics comes into play

State board members have two options: They can accept the cut-off scores, or they can ask to change them if they don’t think they reflect what students should know to pass.

State board members also have another critical bit of information to consider: how many kids would pass and fail based on the recommended passing score.

That’s because the state board sets the cut-off scores after student have taken the exams and all of their scores have been recorded. Next week, they will be setting the passing scores for the ISTEP test taken last spring.

So even after all the work and careful analysis by the educator panels to recommend the cut-off scores, those recommendations can be, and often are, overruled by a state board appointed by the governor and legislative leaders. (The elected state Superintendent Glenda Ritz is also on the state board and gets a vote, too.)

Some board members are educators and might be familiar with the score setting process, but others are not. And they make their decisions without any of the careful steps the educators walk through to arrive at their recommendations.

Roach said the board has the authority to send the scores back for additional shifting. To keep the test statistically valid, they can’t move the the passing score too far.

“The board can give suggestions, but shifting is limited,” Roach said.

Delandshere said she has seen committees change course to avoid failing too many students.

She remembers working with a score-setting committee for a teacher licensing exam. The group had such high expectations for aspiring teachers that it consistently set very high passing scores.

That approach proved to be unrealistic as so few test-takers would have passed.

About 65 percent of test-takers would have failed, Delandshere said, so many that the group reconsidered and went with a lower passing score.

So the exercise isn’t purely academic, no matter what fail-safes are put in place, she said.

Failing students comes at a cost. If 65 percent of students failed an ISTEP test, for example, it would likely result in thousands of kids needing extra help.

“Of course it’s all political,” Delandshere said. “But it’s also a resource issue because if you have such high standards … you have to have resources to remediate.”

A series of judgement calls

Roach said national test companies use this process, as do other states. She doesn’t expect the board members to make big changes. If they did, it would only be after getting advice from the testing company and other experts.

“I would have full faith in them,” Roach said of the experts who support the process. “They are the ones who make sure the numbers all work but that the process is all valid.”

The whole testing process, Delandshere said, from writing questions to setting passing scores, is just a series of judgement calls, whether they come from teachers, state board members or test companies.

“There’s no equation, there’s no rigorous process,” she said. “The best we can do, as I said, is to be inclusive … and to make sure to provide very relevant information and to make people aware of how these test scores and the cut scores are going to be used and the consequences they have for students, teachers and schools.”