The Science of Making Up Stuff

[Catshrink]

This article about the weaknesses of NCLB is from March, but pretty good.

"No Child Left Behind demanded simplicity. All 666 regulation-packed pages of it. It simply demands states set some cut score on some standardized test, declare that that score measures something they must call “proficiency” without any pesky evidence that it means any such thing, then name, blame and shame a school that doesn’t meet its quota of children hitting the cut score, labeling the entire school as failing “Adequate Yearly Progress”, again inventing a term that sounds extremely scientific, objective and quantifiable but which, in fact, means Diddily Poop.

<snip>

I’m good. So I’m mad. Now that it’s time to fix the current abuse of using a child’s standardized test to label an entire school, there’s a move to use a cut score to label an individual teacher.

To do this, of course, they have to Make Stuff Up since there is no pyschometrician, analyst or sober test manufacturer who has ever found evidence that a child’s test is a valid assessment of a teacher. In my class, one student might get an A and another might get an F. Does that make me a C teacher? Well, good teacher evaluation is complicated. Best keep it simple and Make Stuff Up:

Start by using words that sound swell, even though there is no science to defend their use. How about “Effective” teachers?

Pitch: Effective Teachers’ kids have high test scores. Ineffective Teachers’ kids have low test scores. Simple to know who gets the prize. And who gets fired. Pundits can phone this one in.

More:

http://lilysblackboard.org/2010/03/nclb-science-of-making-up-stuff/

[dmallind]

1. So serious question

I admit to knowing little of pedagogy. However I have spent more than a couple of decades understanding how professionals are and can be measured. Many of them are measured not on easily definiable concrete terms such as widget output or sales dollars, but on criteria that demonstrates how they can successfully guide teams, develop subordinates, and essentially work through driving improvement in others (a pretty close analogy to education I would expect). I've seen this done well and I've seen it done badly, but it can surely be done.

So then an obvious two part question arises.

What is it about teaching that makes the ability to measure teachers to reasonable criteria different from other professionals, and how then can teachers be measured? Surely we do not want to see an entire profession where no real external criteria can be applied at all, because that would imply that good teachers and bad teachers are indistinguishable, which would frankly make teachers way less relevant and significant than they surely are. If a skill or profession matters, skill in that profession should be measurable and comparable. I doubt anyone is going to argue that teachers don't matter.

[d_r]

2. my .02

we know, statistically, that there is an "achievement gap" between children in poverty and those not in poverty, and between children of color (who are disproportionally living in poverty and white kids." We see it on standardized tests, we also see it in grades,and retention, graduation rates. There are many understandable possible causes - lack of resources, lower quality schools, more inexperienced teachers in poorer schools, differences in parents' education levels and their orientation towards academics that they stress for the children, institutional racism (including "soft racism" like tracking kids in to higher performing groups and lower performing groups that ends up cutting along racial and economic lines), and many others that come from parents, teachers, communities, and schools. Another question simply is "are tests measuring what they are supposed to measure" - are the tests culturally fair, do they reflect how children learn and demonstrate what they've learned, do they measure what is actually important.

My concern is that when we put too much emphasis on the standardized test scores we may be missing 99% of the story. The scores are convenient to use because we can get our minds around them, compare them to each other, they are straightforward. But there is error in there, and they are certainly not catching everything.
d
What we do know is that there are schools in very poor areas that "buck the trend" of the achievement gap. Where kids are very resilent and they are doing well besides having the same sort of obstacles from poverty. One of the things that we know is associated with those schools is that the students have positive relationships with the teachers - there is a direct link between this and academic achievement. Teachers who are supportive and responsive, in tune with children's individual needs, have students who are doing better. One trick for teachers here is not falling in to a "pedagogy of poverty" and lowering their expectations for children in poverty, and directly or indirectly communicatxing those lowered expectations to children.

The kinds of things that teachers do that are associated with success are things that could be measured directly, and these sorts of dispositions could be taught and rehearsed and acquired by teachers who aren't showing them (but have the motivation to be willing and the open mind). Instead of measuring changes in the children's standardized test scores from time 1 to time 2 and inferring that gains and losses are the result of the teacher, we could measure the teacher's interactions and dispositions directly. That would be more valid.

For example, if I want to know if a Kindegardner could tie his shoes, the best way for me to do that would be to ask them to tie their shoes and I could watch if they are doing it. That would be more valid that giving them a worksheet about tieing shoes and seeing if they did better on the worksheet from time 1 to time 2. It is better to measure things directly, and if you want to measure classroom climate, etc. then do so. Standardized test scores could certainly be one part of that mix, but if we rely on them entirely we are exaggerating their importance, and using them for methods that they were never intended or developed to be used for.

[d_r]

3. really?

I type all that smack and nobody on the whole democratic planet has anything to say? seriously? I am like the freaking thread killer. Flipping internet ortho.