Reply #35: No, it isn't for real. The actual conclusions weren't nearly so dramatic as the story seems to imply

[fishwax]

More importantly, perhaps, the positive results are limited to state tests, while the study itself acknowledges that "many of the gains on state tests are not confirmed by NAEP" (page 70 of the report; emphasis mine). The NAEP is the only national assessment test, and the scores on that test do not suggest improvement in the years since NCLB went into effect. (This is not to say, as the study says, that the NAEP results should be considered more significant than state results, but it certainly counters the enthusiasm some are finding in the study's results.)

The main conclusions are quite generic. I've copied them below from the center's report, which you can find in PDF form here: http://www.cep-dc.org/index.cfm?fuseaction=document.showDocumentByID&nodeID=1&DocumentID=200. I think that there is a lot of spin in the WaPo article, attempting to paint this study as more conclusive and more positive than it actually is.

1. In most states with three or more years of comparable test data, student achievement in reading and math has gone up since 2002, the year NCLB was enacted.

The second page of the WaPo article makes clear that the number represented by "most" varies widely: "In elementary school math, 37 out of 41 states with adequate data showed significant gains. In middle school reading, such increases were found in 20 out of 39 states, and in high school reading, in 16 out of 37." So barely half of the states in middle school reading, and less than half the states in high school reading, showed significant gains in test scores.

There is more evidence of achievement gaps between groups of students narrowing since 2002 than of gaps widening. Still, the magnitude of the gaps is often substantial.

14 out of 38 states showed narrowing achievement gaps. That's less than half. That there was no evidence of widening achievement gaps since NCLB in the remaining 24 states is hardly convincing proof of NCLB's effectiveness at closing that gap, especially since--to the degree that these results are positive--they don't demonstrate causality.

In 9 of the 13 states with sufficient data to determine pre- and post-NCLB trends, average yearly gains in test scores were greater after NCLB took effect than before.

So only 13 states had adequate data, and only 9 states actually showed faster improvement after NCLB took effect, while four states showed slower improvement. The sample size (as the study itself notes) is too small to be anything near conclusive.

It is very difficult, if not impossible, to determine the extent to which these trends in test results have occurred because of NCLB. Since 2002, states, school districts, and schools have simultaneously implemented many different but interconnected policies to raise achievement.

I would also add that (as the study itself notes) there are other factors as well, such as increased teaching to the test, that can add to this. (Indeed, I would question the study's title: Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind?" Actually, what they are answering is "have scores on specific standardized tests increased since NCLB?"

Although NCLB emphasizes public reporting of state test data, the data necessary to reach definitive conclusions about achievement were sometimes hard to find or unavailable, or had holes or discrepancies. More attention should be given to issues of the quality and transparency of state test data.

I wonder if this doesn't suggest a sort of selection bias, in that in situations where scores have not improved, administrators have obscured the data out of fear of losing funding? Just a possibility ...

Additionally, there is another interesting conclusion in the study that contradicts the assertion that NCLB has improved student achievement. While the study shows that scores on state tests have increased, the scores on the NAEP--a national standardized test--have not increased. Indeed, scores on this test have declined in many cases. (And, interestingly, is more likely to decline in the same states where state scores have increased, suggesting that state scores have increased precisely because teacher's--pressured by the high stakes of NCLB--have increasingly taught to the state tests, leaving less time for other parts of the curriculum:

Overall, NAEP trends since 2002 show a less positive picture of student achievement than the state test results reported in this study ... The 2005 NAEP results for grades 4 and 8 indicate that reading achievement has remained essentially flat since 2003 ... during that same period, average NAEP scores also increased in math but at a slower rate than in previous years. Math results for grades 4 and 8 climbed dramatically in the early 1990s, leveled off in the mid-1990s, and then rose again between 2000 and 2003. Scores increased more modestly between 2003 and 2005, when the 4th grade average score increased by 3 points to 238, and the 8th grade average score rose by 1 point to 279. (page 62 of the study; emphasis mine)

In summary: (1) The results are actually quite muted, far from concrete proof of rising test scores or narrowing achievement gaps in the years since NCLB went into effect; (2) There is no evidence of causality between the requirements of NCLB and the changes in test scores noted in the study; (3) While scores on state standardized tests have increased, scores on national test scores have not improved as quickly post-NCLB as they did prior to its passing.