ISTEP disruptions had no impact on scores, review finds

Glitches that disrupted Indiana's online standardized testing this spring had no measurable negative impact on scores, even though at least one-sixth of students taking the exam were kicked offline at some point, according to an independent review released Monday.

Contractor CTB/McGraw Hill reported that nearly 80,000 students in third through eighth grade — or 16 percent of the more than 482,000 Indiana students who took the online test — had their ISTEP exam interrupted at least once when computer glitches kicked them offline during the April-May testing period, the report released by the state Department of Education concludes.

But the problems had no negative impact on scores for the "vast majority" of students, said Dr. Richard Hill of the National Center for the Improvement of Educational Assessment Inc., who conducted the review. Hill said, however, that 600 students who were taking the test lost data "that was not 'restored' when they logged back in."

Student ISTEP scores slightly increased between 2012 and 2013, on par with improvements in previous years, Hill said.

"It does appear, looking at the data, that the net impact of the interruptions was nil. Students scored about as well as they would have had no interruptions occurred," he told Indiana's Commission on Education on Monday. "That may come as a surprise to many, but there's a lot of evidence to back up that statement."

Hill credited school staff's actions as well as students' own resolve to press on and work "diligently" to complete the multi-part test.

State Superintendent of Public Instruction Glenda Ritz halted testing for two straight days in April as the server problems knocked students offline or left them unable to access the online test, which assesses students' proficiency in mathematics and English/language arts. Some students are also tested on their proficiency in social studies or science.

Because of the problems, the state asked schools to cut their normal test loads by half for a few days and the testing period was extended.

Hill's report states that school districts reported an additional 60,000 students' tests were interrupted by the glitches. That figure, combined with the number reported by CTB/McGraw-Hill, would mean 29 percent of students tested were affected.

Hill said he considers the company's numbers more accurate than the schools', but added that the school's numbers cannot be discounted.

Ritz told the commission Monday it's unclear how many students' tests were affected in some way by the disruptions.

"You need to keep in mind, anyone sitting next to somebody who's interrupted for any length of a period of time, that's going to cause an interruption in their own testing, whether they didn't spend time on the question they were on, or whatever the case may be," she said.

Indiana's troubles punctuated a nationwide shift from pencil-and-paper tests to online exams. Troubles earlier this year also disrupted high-stakes testing in other states, including Kentucky, Minnesota and Oklahoma.

In Indiana, ISTEP scores are used to assess student performance and determine teacher pay and school rankings.

Ritz said the testing problems were "extremely stressful" for teachers and students. She said that given the uncertainties about how students might have scored without the glitches, she's given local schools the "flexibility to minimize the effect" of test results on this year's teacher evaluations.

"There is no mandate. It's up to the individual school districts to look at that," she said, noting that 2013 is the first year Indiana is tying student test scores to teacher evaluations.

Ritz said the state Department of Education is continuing to negotiate with CTB McGraw-Hill to seek a settlement with the company, which has a four-year, $95 million state contract to administer the test.

In June, Ritz said her office was seeking at least $614,000 in damages from CTB/McGraw-Hill for the testing troubles. That amount included $400,000 for fines covered in the company's state contract. The remainder would pay for the independent review and better reporting data.

Schools typically receive ISTEP results in May, but Ritz said the glitches and the review means the data won't be released until late August.