Suspicions about 2005 test gains now confirmed

by Paul Socolar

When the District made the announcement of a big increase in its fall TerraNova test scores in December 2005, the Notebook started asking questions.

Just months before, School Reform Commissioner Dan Whelan had challenged the wisdom of an SRC decision moving the annual TerraNova test in grades 3 through 8 from the spring to the fall. He said assessment experts had told him that the results on a test given in the fall would not be comparable to results from the prior spring.

The Notebook found that some testing experts shared that concern about the nationally normed test. There were also questions about whether students who had taken the TerraNova test twice within a six-month period might do better the second time simply due to a “practice effect.”

Publicly, District officials and CTB/McGraw-Hill, which produces the TerraNova, insisted that the increases in student scores here between spring 2005 and fall 2005 were real gains and that there was no problem comparing fall results with previous spring scores.

When the Notebook posed its questions about the scores to assessment professionals on the Accountability Review Council (ARC), an independent state panel, the District issued a five-page memo along with its testing company, defending the validity of the Fall 2005 scores and of the comparison with earlier TerraNova results.

But when Philadelphia's fall 2006 TerraNova results came in much lower than in fall 2005, the District changed its stance.

In February 2007, District Chief Accountability Officer LaVonne Sheffield disclosed that her office is investigating whether a practice effect or some other outside factor caused a spike in the 2005 results, and she directly affirmed the Notebook's line of questioning. The District hired two national testing consultants to examine the issue. There is no evidence of cheating or tampering, Sheffield said.

CTB/McGraw Hill is conducting its own analysis but still says a six-month window for retesting should not be a problem, even though half the test questions remained the same.

While no definitive explanations have been provided, both the District and the testing company now discount the Fall 2005 scores as an “anomaly” – inconsistent with all other District test results.