Indiana’s standardized tests might have gotten so much harder last year that test scores plunged across the state, but two national testing experts say the exam might technically still be too easy according to an earlier study.
“The Indiana test was relatively low-level,” said Ed Roeber, a former Michigan testing director asked to consult on ISTEP.
The 2015 ISTEP exam might have seemed tougher to students and teachers. The state saw a 20 percentage point drop in students passing both English and math last year.
But when Roeber and another testing expert, Derek Briggs, were asked by the Indiana State Board of Education to conduct a review of that exam in January, they discovered that exam questions were not as rigorous as they should have been. The state adopted new academic standards in 2014 that were designed to better prepare kids for college and careers. The new standards were said to be much tougher.
Indiana officials brought on Roeber and Briggs on to advise them on the ISTEP last year in the wake of scoring glitches, delays and accuracy questions that raised concerns about the validity and reliability of exam. After examining an earlier study of the test by the education research group WestEd, the pair noted that Indiana’s test questions do accurately reflect the standards, but suggested that they might still be too easy.
The analysis from Roeber and Briggs reached no conclusions on whether the standards themselves are meeting the state’s goals, but the pair found that too many questions on last year’s ISTEP test were too easy.
Specifically, the experts cited the WestEd study that found a majority of the questions on the 2015 exam were fact-based questions that asked students to merely to recall information rather than do higher-level thinking. More than 80 percent of questions on the English exam and every single question on the math test were found to be low-level basic questions.
The test had very few more difficult questions that would have required students to describe or explain a scenario, use evidence to back up answers or test a hypothesis and make connections beyond the facts at hand.
“The fact that (a test question) aligns with the standards is good, but it doesn’t imply that it’s measuring the standards to the same depth the standard is written at,” Roeber said.
What remains unclear is why the questions were too easy. The fault could rest with the test, which was written hastily after lawmakers voted to drop the Common Core and write Indiana-specific standards in 2014. Test questions weren’t tried out on students until the test was actually given last spring — a highly unusual practice in a testing industry in which test questions are typically piloted over several years before being used on an exam, Roeber said.
The Indiana Department of Education decided against using questions from previous tests that could have saved some time, said Cynthia Roach, the state board’s testing director.
Another possible reason for the easy test questions is the state’s standards themselves. It’s not clear just how rigorous Indiana’s standards really are. Until the standards are properly evaluated using the same metrics as the test, Roeber said, the state’s policymakers can’t really know if ISTEP is measuring what kids are expected to learn better than past tests.
It might seem technical, but the mismatch between the standards and the questions could be important. Indiana still must submit its test for review by the U.S. Department of Education for its waiver from the No Child Left Behind law, still in effect until August. Any misalignment between the test and standards could be problematic.
“That’s one of the key things that the peer reviewer will look for is the measure of the rigor of the test, and if this misstates that, then you definitely have the right to challenge the vendor,” Roeber said.
The state’s new testing advisory committee is taking the suggestions from the analysis to try to move forward, Roach said. Nothing can be addressed in time for the 2016 test, which students have already started taking, but questions should be reassessed and Indiana should look into deeper measurement of its standards, Roeber said.
The Indiana Department of Education, which administers the test, declined to make its testing director available for an interview to discuss Roeber and Briggs’ analysis, but Department spokeswoman Samantha Hart issued a statement saying the test was hard enough.
“As with any assessment, there are going to be questions that are more and less rigorous than others — The new ISTEP+ exam is no different,” Hart wrote. “Anyone who thinks that last year’s ISTEP wasn’t hard enough should go talk to a student or a teacher or a parent. This test was clearly more rigorous, just like our standards.”