I like the idea of asking the kids:
How useful are the views of public school students about their teachers? Quite useful, according to preliminary results released on Friday from a $45 million research project that is intended to find new ways of distinguishing good teachers from bad.
Statisticians began the effort last year by ranking all the teachers using a statistical method known as value-added modeling, which calculates how much each teacher has helped students learn based on changes in test scores from year to year. Now researchers are looking for correlations between the value-added rankings and other measures of teacher effectiveness.
value-added modeling:Research centering on surveys of students’ perceptions has produced some clear early results. Thousands of students have filled out confidential questionnaires about the learning environment that their teachers create. After comparing the students’ ratings with teachers’ value-added scores, researchers have concluded that there is quite a bit of agreement.
In value-added modeling, researchers use students’ scores on state tests administered at the end of third grade, for instance, to predict how they are likely to score on state tests at the end of fourth grade. A student whose third-grade scores were higher than 60 percent of peers statewide is predicted to score higher than 60 percent of fourth graders a year later. If, when actually taking the state tests at the end of fourth grade, the student scores higher than 70 percent of fourth graders, the leap in achievement represents the value the fourth-grade teacher added.
I know all professions are vulnerable to what I think of as “fads” and teaching seems to be particularly defenseless when presented with magic bullets and over-hyped theories.
Is value-added modeling just another loser in a long line of teacher evaluation theories, or is there anything to it?
Punchy
As a teacher, I’ve learned one universal thing: if you give the student an A, he/she will almost without fail give you great feedback/comments. If you give them a C, you’re their shittiest teacher EVAH. B students dont bother replying at all. Goddamn cudlips.
kay
@Punchy:
I was a horrible student in high school, generally, so I don’t feel qualified to measure.
I wouldn’t have been there to take the survey :)
Yutsano
Nope. Calling foul. And this is why: it gives all the credit to the teacher for learning that could have happened from a variety of sources, including the student’s own initiative. By that model, my teachers would have experiences multiples in the high fifties if not higher, but the vast majority of that I learned from doing myself. Unless there is a method for smoothing out the edges and taking outside sources into account, it’s giving credit where it’s not due.
Marcelo
It’s not a complete tool, since so much can affect whether or not a kid does better from one year to the next. Certainly a great teacher can push a kid past where they normally would go and a terrible one can drag the kid down, but it’s just as likely that parents divorcing, parents losing their jobs, a bully in the classroom, drama wrt boyfriends or girlfriends or whatever, the kid running with a new group of friends that brings on negative choices, all these things and more impact how well the student will do.
The reasons why kids perform the way they do in school is INCREDIBLY chaotic and impossible to pin down, much like the economy. Value-added analysis is useful to a point, but the problem arises when it’s used to rationalize someone’s very political firing or rehiring. The administrator doesn’t like the teacher anyway and uses the VA report as a crutch.
Dennis SGMM
The flaw I see in value-added modeling is that it assumes that the scores on state tests are a measure of whether or not a student has learned to do anything more than take a state test.
Depending on the state, I’d guess that some youngsters who passed their state tests with flying colors would get clobbered on the SATs or on their first Upper Division test that included essay questions. I’d love to see some actual data on whether or not someone’s scores on state tests are in any way a predictor of their success in college or in life.
Punchy
@Yutsano: Are you saying that 3rd graders take it upon themselves to learn social studies, math, and history outside the classroom? Maybe 1% would do this. A vast majority of 9 year olds are playing Wii or getting noogeys from their siblings outside of the classroom.
Davis X. Machina
Cum granō salis, people: A student’s fifth grade teacher was an excellent predictor of his or her fourth grade reading and math gains…
See Jesse Rothstein, “Teacher quality in educational production: Tracking, decay, and educational achievement”, NBER Working Papers 14442, National Bureau of Economic Research, 2010
Phyllis
If a kid had the same teacher from K-12, then this might be viable. Like Marcelo said, there are so many variables from year to year-the overall makeup of the class (new kids in the mix), change in state standards, new/revised textbook series, new grant funds available, or a grant that has ended. Add to that that sometimes the kid and the teacher just don’t click. While a lot of it is about the teaching, it isn’t 100% about the classroom instruction.
Zam
THERE IS SO MUCH SNOW! My car disappeared.
burnspbesq
This is what happens when you try to measure things that are inherently unquantifiable. There are more variables in this equation than anyone can reliably identify, and God help us all if anyone tries to do a regression and use the r-squareds to determine funding priorities.
Menzies
@Dennis SGMM:
This, pretty much. Value-added modeling is bullshit.
The reason teachers are “defenseless” is because teaching is such a vague collection of psychological factors that it’s really easy to just blame it on the educator rather than take a look at any other standards you may be pushing.
As a sort of hypothetical, if TX had passed all of its new changes to science and history, in twenty years, their students would suck in both subjects, and guess who would get the blame? Teachers.
Omnes Omnibus
@Punchy:Between 3d and 4th grade my reading speed jumped by quantum levels. Prior to that, I preferred having someone read to me because I could picture the story happening in my mind. Once my own reading speed got fast enough that I could see movies in my head as I read, I read all the time; I read everything. I learned a shitload outside the classroom. My 4th grade teacher had almost nothing to do with the fact that clicked in my just turned 9 year old brain.
burnspbesq
@Menzies:
Vastly overstated. It’s a potentially useful tool, but not when used all by itself.
In theory, the best way to evaluate teachers is by observation in a real classroom setting. Alas, that runs smack into Heisenberg, because the presence of an observer changes the behavior of the observed. And if the people in the classroom think there might be a hidden camera, they will behave as though there is one, regardless of whether there actually is.
Joel
The “value added” sounds like normalization. That makes it one big (logical) step better than just making judgements based on bulk standardized testing scores.
Davis X. Machina
Cum granō alterō salis: “…only about a third of teachers ranked in the top quartile of value-added based on one academic year’s performance would appear in the top quartile again the next year. And ten percent of bottom quartile teachers one year would appear in the top quartile the next.”
Brookings Institute Report: Evaluating Teachers: The
Important Role of Value‐Added
Loneoak
Again we see a moral fetish with ranking “good teachers.” Next year, every teacher could be better than every other teacher that has previously existed in the history of humanity and yet 50% of them would be below average teachers. We could, as a thought experiment, give every 3rd grader a PhD in mathematics or theoretical physics and still 50% of them would be below average students with below average teachers.
This is not to deny there aren’t some pretty awful teachers struggling in impossible situations, but ranking them won’t do a damned thing about it. The question is whether our childrens is learning, and whether they are learning the right material in a supportive environment, not whether they are sufficiently ranked against a mean.
sublime33
Full Disclosure: My wife was a teacher, actually a reading consultant responsible for raising reading scores for the elementary school district. As a financial data analyst, I always argued that using individual student change was vastly superior to using the commonly used gross school test scores. Total test scores are a lazy measure that is grossly biased by the education level of the school population. Any administrator with pulse should be able to keep a wealthy school’s scores above 80%, and the greatest superintendant in the world is going to have a tough time getting the poorest school above 25%. So a built in excuse is handy for both the wealthy and poor school administrations. And the teacher’s unions will fight it across the board because they don’t want a new reason to be reprimanded or dismissed, especially since there are so few reasons today.
The one flaw in using this method is that it should be understood that wealthy schools are going to start out with a huge head start with a lot of 2nd graders scoring in the high 90th percentile. But as they get older, middle income students close the gap. So the expectation should be that a 6th grade teacher in a wealthy school is going to see a lower percentile rank than when the same students were in 4th grade. Plus, they have a lot more room to fall down than drift up. Likewise, middle income schools should expect a drift upwards as they get older.
ChrisZ
Value-added evaluations aren’t bullshit. They are flawed, as is any evaluation method. Value-added absorbs the flaws that exist in whatever test you’re using for an evaluation (and the US doesn’t give very good tests), and can’t determine what the actual causes of differences in performance are for any given student. Of course, statistical methods aren’t meant to work for any given individual.
But the existence of flaws doesn’t make it bullshit, or worthless. A teacher whose students consistently improve according to the value-added modeling is probably doing something right. Maybe some kids did it on their own, or for other reasons, but if a bunch are doing it consistently we might think the teacher has something to do with it. We should probably try to identify those teachers and figure out why they are successful.
What I think a lot of people who react strongly against a model like this are really saying is “You can’t reduce teaching to a number.” That’s true, and important to remember. I would never suggest that a value-added model should be used to, by itself, determine teacher salaries or school funding. It’s a common human tendency to overvalue things that can be reduced to one simple number, and undervalue more subjective evaluations, so there is some reason to be cautious around numbers. But it’s a useful tool, among other useful tools, and I don’t think it can be written off as worthless or bullshit just because it isn’t perfect.
Belafon (formerly anonevent)
Most people’s reaction is going to be the same reaction I saw when my last employer started rating employees. Most people thought they were above average, and so when you rated everyone, they were going to get screwed. Once again, most people thought they were above average.
MattR
@Dennis SGMM: @Marcelo: I think these two comments sum up the problems with value added modeling pretty well. However I still believe that if you take those flaws into account, value-added modeling can add value to the way that teachers are evaluated. As a very simple example if there are two fourth grade teachers in the same school but the students in one teacher’s class are consistently improving more than the students in the other class, it is probably a good idea to investigate things further. The two teachers may be equally good but one teacher is “randomly” being assigned “better” students or maybe one teacher is focused on teaching to the test. Or maybe it is the case that one teacher is actually better at their job than the other. But either way, it is probably a good idea to identify the cause and see what lessons can be learned.
Davis X. Machina
@ChrisZ:
In other words, you’re not a superintendent of schools, or a state legislator….
SpotWeld
It’s one thing to use studies and surveys like this to get a better understanding of the state of a given school… or maybe even gage how things have changed over time.
But they are utterly toxic when they become a benchmark or a quota to use and the main factor in various hiring or salary decisions.
Information should be something that helps you think, not something that eliminates the need to think.
(I have the same problem with “zero tolerance” policies. Value = x, then do action B. No thought needed)
gelfling545
@Yutsano: I have to say that I have seen this in action. The first year State testing was done in my district by some weird chance we had a group of really amazing students in 4th grade and got the second highest scores in the city – the honors school being first. Again when these students were in 8th grade, same thing, but here’s the problem: I knew these students well and if we had left them alone for 4 years they probably would have done just as well. We couldn’t take the credit but people want to give all credit or all blame to teachers and it really doesn’t work that way. Students are people not subjects to be acted on.
Mnemosyne
@Punchy:
I see you were not a nerdy kid, because I was doing all three of those things. In fact, they had a hard time teaching me long division in 4th grade because I had read about negative numbers over the summer and the teacher wouldn’t (or couldn’t) explain to me why I couldn’t use them in long division to get the number I wanted. “Because I said so” has never worked well for me as a method of education.
bemused
Observing over the shoulder of a very good new teacher in a poverty school, I got to watch her outrage as she discovered that the school administration encouraged cheating on the test. Some teachers succumbed to this pressure, and some didn’t. So although my tendency before this experience would have been to applaud value-add metrics, I would now be very suspicious that a good score on such a metric was selecting for dishonest teachers.
ChrisZ
@Davis X. Machina:
I cannot deny that that’s true.
cthulhu
As Loneoak notes, using a distributional change approach might work if the only factor was the teacher but even then, as one starts to weed out teachers by ability the distribution becomes more and more about random noise. So inherently the model is incorrectly specified. Not to say that it could have some use but one wouldn’t want to rely on it. My statistical experience in not in Education and they do utilize some tools that I don’t use all that much but I think there are far more technically sophisticated ways to approach the problem.
Also, they are talking about within US comparisons. Going forward, it would seem we need to turn our focus to international reference points if we intend for our kids to be truly competitive in the future.
Dennis SGMM
@MattR:
Yes, that’s why I didn’t write that it was completely useless. Yes, teachers should be evaluated and, just as good teaching is nuanced and adaptable to the composition of each new class, the evaluation of teachers ought to be the same. I become nervous when anyone trots out a Silver Bullet – especially when the last one, NCLB, has left our kids down around twentieth in math and science worldwide.
Davis X. Machina
@ChrisZ: Most people don’t need there to be any firearms laws. That doesn’t mean we don’t need firearms laws.
Jim Pharo
Why is teacher evaluation so much harder than evaluating any other worker?
The “value-added” evaluation may well have some utility as a tool, but can you think of any other job where we pretend evaluation is so tough?
To me, this is a 100% gold-plated distraction. The challenge facing our kids is not that we can’t tell good teachers from bad. It’s that we have given up on having a system of actual education and are content with just killing time.
The fact that we talk a big game about education is, to me, compelling evidence that We All Know(tm) that our education system is basically a cruel joke.
Karmakin
Yeah read #15. That’s pretty much all you need to read. I wouldn’t go so far as to say it’s bullshit..in terms of trying to find potential problem areas, it’s better than pretty much every other statistical method, but the problem is that you’re trying to model something that’s insanely complex and has way too many variables to account for. It simply doesn’t work, and it shouldn’t be used for anything other than an innocent flag. As in a teacher who is always at the bottom, you might want to do some investigation on the circumstance, their teaching style, the student base, whatever, and find out what’s going on. But the evidence shouldn’t be used to fire the person or slash their wages.
In any case, to be honest the efforts to improve education are mostly a fetish desire to try and fix the economic issues without actually taking a step forward and recognizing that it’s the overall structure that’s broken, and not necessarily individuals. Now, that means that the middle class/upper class individuals are not really special snowflakes who deserve oral sex on the hour. That’s why we spend so much time worrying about this stuff.
ChrisZ
@Davis X. Machina:
Sorry, but I’m not following the analogy. It’s possible that studying for exams has fried my brain.
Davis X. Machina
@Jim Pharo:
It might be worth reflecting, therefore, that the Brookings Institution is ready to roll with VAM, on the grounds that it’s no less accurate than SAT and ACT tests in predicting student academic success in college.
We know how well that sorting system works….
PeakVT
Teachers need to be evaluated, but I think all the talk about testing and rating and performance distracts makes it seem like teachers are the only factor relevant to student performance. Clearly they are not. I think the focus on teachers is part of the conservative war against government more than a genuine attempt to improve overall student performance in the US.
Davis X. Machina
@ChrisZ: I wouldn’t misuse them — I wouldn’t go near them — and you wouldn’t misuse them, but they get misused all the time, with predictable and disastrous results. And then there are those who have made a fetish of them…
Karmakin
@Jim Pharo: There is no real problem with education. Ok, there are problems with education in the American system, but most of them can be fixed by not having schools funded generally on a local basis.
But there’s no shortage of workers, there’s no shortage of educated workers (in an ever-growing number of fields). There’s simply not. America (and the rest of the 1st world) is growing less competitive because our wages are too high in comparison to the second world.
It’s all about lowering, not raising wages.
MattR
@Mnemosyne: I remember having similar discussions with my math teachers in junior high (though I think it may have had to do with the shortcuts they were trying to use to explain exponents). The district ended up “suggesting” to my parents that I take an alternate path to studying math to make the teachers lives easier. End result was that I spent about a month in the summer after freshman year completing 11th and 12th grade math, I took no math in 10th grade, took calculus in 11th grade and then took differential equations at the local community college during 12th grade.
FormerSwingVoter
I’m very skeptical of putting teacher evaluations in the hands of their students; however, value-added modeling sounds like it… has promise, anyway.
Lots of different factors affect how well kids learn, but that’s true (albeit to a lesser extent) with most methods of evaluating anyone’s work performance.
Nylund
I actually know a thing or two about this stuff.
1. The alternative to “value-added” is to simply look at one year’s score in isolation. Joe got an 80. Bob got a 70. What do you conclude? the teacher, school, peers, parents, or whatever for Joe are better! Obviously that is dumb. But, if you know that Joe got a 90 last year and Bob got a 50 last year, then you can say, Joe did worse than usual, but Bob really improved. In essence, you want to find out what factors lead to improvements and which don’t (or make things worse). Think of it this way, if I told you that Bill went on a diet, and after his diet, he weighed 180 lbs and asked you if the diet was effective, the first thing you’d ask was “well how much did he weigh before? Same thing here. You have to know how the student did in the past before you can figure out if there was improvement. That “improvement” is what is meant by “value added.” If a B student stays a B student, then no value was added.
2. Yes, test scores are a crappy measure of what was really learned. Unfortunately, data sets don’t have a variable called, “what was actually learned.” You do the best you can with the data you have. EVERYONE who does this type of research knows test scores are merely a proxy for what you really want, and probably a pretty crappy proxy at that. Very rarely is there actually an empirical measure for the thing you actually want to measure. You do the best you can with the best you’ve got (and think long and hard about the implications of it being a crappy measure).
3. When you look at something like the affect of teachers on students (or funding, or computers, or class size, etc.) you are NOT attributing all the value added solely to that one thing. We all know that bullies, parents, income, what neighborhood you’re from, if one of your parents died or got sick, parental educational background, gender and racial compositions of the class, school, etc. school funding, teacher experience, etc. all may or may not have effects too. Not only that, they may have non-linear effects, or may interact with each other.
Some of this you’ll have data on, some you won’t. Once again, you use the best information you can get, but no data set contains everything you want to know. These factors all goes into the analysis and the goal is to find their effects too so you can properly separate out the effects of one thing from another.
In short, there are a ton of potential problems. Measurement error, omitted variables, selection bias, endogeneity, etc. etc. Researchers know this and do the best they can with the data they have. There is nothing wrong with a “value-added” approach. The question here is how reliable data from children are. As an earlier comment points out, if you give a student an A, they will give you a good evaluation. If you use that as your measure of quality of teachers, then you end up with good grades being the cause of high teacher quality (rather than the effect). I’d wager you have a major endogeneity problem using student evaluations.
ChrisZ
@Davis X. Machina:
Yeah, I can see where you’re coming from. Perhaps I’m just more optimistic than you here, but when a useful tool comes around I like to try to use it properly rather than not use it at all because it might be misused. That’s why I generally don’t mind government programs and regulation, they might cause more harm than good, they might cause more good than harm, I like to try for the more good than harm outcome rather than give up.
But we’re really both just guessing about whether it will do more harm than good. It’s not like our current system is terribly successful though. Evaluating teachers based on a once-a-year observation, test scores that ignore where the students started, and the number of complaints received isn’t a very good system to start with, and I’m not even sure that using value-added as the be-all-and-end-all would actually be much worse. Some teachers would be treated unfairly by it, but would it be more than currently are?
cthulhu
@Jim Pharo: Actually, I would say that evaluating teachers is a more complex problem than evaluating the average individual employee. It is more akin evaluating a manager or, um, CEO. The output of a teacher is via the performance of other individuals (the kids) and there resources are often out of their control (and are often derived from largely unchangeable economic disparities). And assessments of managerial performance are often a joke.
Frankly I think the curriculum is of most importance. Once that is optimized, then you can focus upon how well the teacher is delivering it, getting students committed to it, etc.
Davis X. Machina
@Nylund:
It’s not a wager if the outcome is not unknown. And this outcome is perfectly known — check out university instructor-evaluation systems.
Cat Lady
All teacher evaluations will do is confirm the obvious – educated parents will make sure teachers educate their kids too. There will be an economically disadvantaged kid here and there that will respond to a good teacher in a way that may change his or her life, but otherwise without a home environment that values education, teachers’ ability to make a substantial difference is extremely limited. Haven’t you seen Season 4 of The Wire?
Marcelo
Value-added modeling to me is best used to evaluate STUDENTS, not teachers. Data-driven processes that can identify students that are falling behind and then direct resources (tutors, mentors, counselors, parental engagement) are invaluable because so many times the kids that fall behind also fall through the cracks, and data driven processes can flag those students who don’t otherwise get attention because they’re not loud enough or whatever.
But using VAM to evaluate teachers? That’s so tough. I know amazing teachers whose students totally HATE them, resist all attempts at learning, all indications are the teacher stinks, but then it’s like Roald Dahl’s delayed action mousemaker in The Witches – three years later something that hardass teacher said clicks and the kid’s worldview shifts a little subconsciously and suddenly they’re responsible kids who do well and pick themselves up. Even if you’re a hateful disciplinarian, at some point you’ll have a kid who didn’t learn a damn thing in your class come up to you ten years later and tell you that you changed their life and that you are the reason they avoided prison or whatever.
Good teaching isn’t just about curriculum, standards, and making sure the kids knows shit. It’s also about modeling behavior, modeling success and responsibility and all that, and SO MUCH of that is subconscious and doesn’t necessarily happen the same year you teach them.
jcgrim
For a list of research on error rates in value added assessment and standardized testing see
Value added scores are unstable and unreliable indicators of teacher quality. According to EPI literature review:
From DoE’s research synthesis:
Secretary of Education, Bill Gates and his assistant Arne Duncan have mandated teachers jobs should be contingent on value-added scores. Remember when Bush’s EPA ignored EPA scientists warnings about climate change? Same song different department.
Brian S (formerly Incertus)
I don’t know from modeling, but I wonder why anyone would ask students, who in general know shit about shit especially at a young age, what kind of job they think their teachers are doing. My college students don’t know what kind of a job I’m doing, because they’re not experienced enough to know what a good learning environment is. Most of them would rather have classes piped into their laptops while they sit in their residence halls than actually go to a class and interact with a live human, and many would argue that the former is a better learning environment even though it’s decidedly not the case.
Violet
@Jim Pharo:
What other industry has workers whose job is to get a set of non-adult individuals, who are growing and changing in a large variety of ways, to produce a certain set of mental outcomes? None that I can think of.
Businesses are predicated on adults working with each other. Even colleges and universities have adults teaching adults, even if the students are barely across the adult threshold. What other industry requires adults to get non-adults to produce outcomes? Sweatshops? Repetitive motions are not the same as analysis and synthesis.
Kids are growing and learning and changing in a variety of ways that often have zero to do with what happens in school. For example, a kid may hate reading. Suddenly he or she discovers science fiction from a book they stumbled upon at their cousin’s house and now they love reading. But only science fiction. However, their reading scores in general improve. Is that because of the teacher? Probably not. How do you evaluate that teacher, then? Who’s even the teacher? The cousin who left the sci-fi book lying around?
cthulhu
People forget that the original intention of IQ tests were to identify learning deficiencies in students and were never meant to rank those of normal and above mental ability. Likewise, I would think that any teacher rating system should be designed to root out inherently bad teachers (and either fire them or correct their deficiencies, if possible) and otherwise not be used to reward those performing at or above criteria.
Davis X. Machina
Fed special-education law is driving schools in that direction, under the rubric of Response-to-intervention.
jcgrim
David Sirota suggests edu-reform is a red herring used to obfuscate failed neoliberal economic policies. Value-added measurement is one of an entire package of schemes to make privatization of public schools lucrative. There is no doubt the edu-capitalists want to loot the public coffers. Check out the 2nd annual ski summit:
For a list of research on error rates in value added assessment and standardized testing see
Value added scores are unstable and unreliable indicators of teacher quality. According to EPI literature review:
From DoE’s research synthesis:
Secretary of Education, Bill Gates and his assistant Arne Duncan have mandated teachers jobs should be contingent on value-added scores. Remember when Bush’s EPA ignored EPA scientists warnings about climate change? Same song different agency.
jcgrim
Oops the ski summit link is missing from the previous:
http://us2.campaign-archive.com/?u=cfb5b3dae0d0b7da2e949d8d1&id=078f7efb22&e=2e4c6c94e1
Note: Wireless Generation is Rupert Murdock’s latest education acquisition.
Link for list of articles on value added is here:http://www.stickwithanose.com/2010/10/25/statement-on-test-based-approach-to-teacher-evaluation/
Is there something wrong with the link tool?
Mnemosyne
@Brian S (formerly Incertus):
I have to disagree with you — I think kids know who’s a good teacher and who isn’t. I had a crappy teacher in 6th grade who thought English class should consist solely of diagramming sentences. I had incredibly poor math instruction pretty much all the way through school — telling me to go home and do the worksheet for stuff I already didn’t understand wasn’t really teaching me very much. Young kids understand who’s a good teacher and who isn’t much more than you think.
You’re probably right about high school and college kids, though.
sven
@Davis X. Machina: @MattR:
Getting an effective measure from this sort of model is going to be really, really hard.
The simple thought experiment I keep working through is this:
I assigned a number to the portion of classroom learning that is captured in a set of student test scores x. (0=nothing, 1=perfect)
If I assume reliability for both the test and the value added model is .75 then I can say our measure of total classroom learning looks something like this:
(.75)(.75)(x)
As x decreases the measure becomes very weak very quickly. If the standardized test was awesome to begin with, the VAM would still be a flawed measure of teaching quality. If the validity of the test is questionable then the measure of teaching quality is extremely weak.
(ok, where did I make my mistakes?)
Dervin
Oh Dear God, the statistical idiocy in this thread makes the baby Jesus cry. To make it a simple as possible:
If one student goes from the middle percentile to the top percentile, there’s a whole host of reasons why this could happen, biological maturity, improved home life, psychological maturity, good teachers.
If the test scores of a class of thirty students goes from the middle percentile to the top, the common attributes shrink to one: Teacher skill.
And the same could be said for a decline.
What does Value-add give us, a nice framework for identifying further areas of study.
Whenever a professional says their work is too complicated to be evaluated, they are pulling off a snow job of Damien Hirst proportions.
Davis X. Machina
@sven:
Not playing hoops with the President, is my guess…
Brian S (formerly Incertus)
@Mnemosyne: Nope, and I don’t think you’re going to convince me otherwise on this. Kids don’t have the necessary experience to judge quality, not even smart ones, especially not when they’re in the middle of a class. I wasn’t a good judge of a quality teacher until I was in grad school, and even then I got it wrong on occasion, and I was in my thirties by then. No way is a fifth grader competent to judge what’s going on in a classroom. The very best they can give is feelings and perceptions, and that’s of limited value at best, because some of the things they may dislike (like diagramming sentences, for example) may be of some real utility down the road.
sven
@Jim Pharo: Value added models have been most effective when the outcome measure is easiest to agree upon. If you are a factory worker making widgets then analysis of an individual’s contribution to ‘team’ widget production looks reasonable. Likewise law firms can model revenue generation for partners because dollars are so easy to measure. Teachers don’t have a bottom-line output that is easily quantifiable.
sb
@Nylund: Wonderful post.
Regarding this: “EVERYONE who does this type of research knows test scores are merely a proxy for what you really want, and probably a pretty crappy proxy at that.”
Yeah, but EVERYONE who wants to bash teachers and their unions (a growing crowd) are using VA as “the best evah” evidence that teachers should be fired.
So yeah, I agree that VA can be a great tool to evaluate the job I’m doing. But that’s not how it’s going to be used. The current narrative in education where I live (Los Angeles) is that we teachers, with the help of our union masters, are getting in the way of “reform”, and are doing everything we can to avoid an honest evaluation. That narrative is being driven by my hometown paper. And while VA may not be bullshit, the narrative is. We are publicly paid employees working in an industry that is largely seen as a failure and one that is notoriously difficult to evaluate. So naturally, the outrage meter for the uneducated electorate is at Defcon 3.
Brian S (formerly Incertus)
@Dervin:
Or that the teacher cheated on the standardized tests, which has been known to happen. And a previous teacher cheating on the tests can cause a fall in those scores as well. Again–more complicated than almost anyone wants to admit.
sb
Mnemosyne: “Young kids understand who’s a good teacher and who isn’t much more than you think.”
I’m a teacher. I wouldn’t be comfortable with kids playing a part in my evaluation and I’m considered pretty good. But you’re statement is correct–they know. And they should be heard. For every asshole student who gives a bad evaluation out of spite driven by lord knows what, there are ten who answer honestly.
ChrisZ
As far as the students knowing which teachers are good goes, I think there’s an element of self-fulfilling prophecy there. The teachers students think are good get more respect and attention, which in turn makes them better.
Mnemosyne
@Brian S (formerly Incertus):
I have to say that diagramming sentences was never of real utility to me, and I have two writing-heavy degrees (Cinema-TV Critical Studies and MFA in Screenwriting).
If you’re a smart kid, and you know that the information that the teacher is giving the class is flat-out wrong, does that mean that you just don’t understand the secret brilliance of that teacher in giving her students wrong information? Or is it just barely possible that what you’re dealing with is a crappy teacher?
Martin
@Yutsano: Doesn’t matter.
Assessment is a big part of my job. Here’s what you want to look for:
1) Direct measurement of student achievement independent of the instructor. Grades actually don’t matter – they’re too arbitrary from instructor to instructor and they (rightfully) measure things like effort, etc.
2) Ensuring that incoming achievement is properly accounted for.
3) Use the broadest reasonable standard – national, state, etc.
In K-12, you usually want something like this:
Measure student achievement in each subject when they enter each grade. That sets the baseline on entry and doesn’t penalize teachers for the deficiencies of previous teachers.
From that, refine your exiting outcomes – what you’ll measure by the end of the grade. These are usually standardized for advancement, but you might want to set higher standards if the students are strong coming in. These standards don’t need to be all or nothing. Standards for advancement usually say that every student must do x, y, z, but a higher local standard might say 80% of students should be able to do a, b, c. No penalty for having some students that are struggling, but if a lot of your students will easily reach x, y, z on entry, there’s nothing wrong with demanding that teachers aim for a higher target. What we really want to have happen is that students make a reasonable degree of progress from the start to the end of the year.
Use individual classroom progress in the context of a broader statistical base (national is ideal but it could be state or county or city or district, depending) to determine whether each student population made expected progress.
Bottom line on teaching is that we really don’t care all that much how the students got from point a to point b (we care intensely if they don’t get there, however). If teachers motivate parents and get learning to happen extracurricularly – that’s fine. If it’s all in the classroom – that’s fine too. It’s not that we necessarily want teachers to take different paths to the same goal, it’s that students often need to take different paths. They learn differently. We want teachers to be flexible and creative in teaching and if something doesn’t work for half the class, we don’t want them to keep doing it just because some commission said so. We want them to try something different.
This method allows schools to also document performance over time (which is what also matters). With different groups of students each year, a difficult or easy group will average out for each teacher. And to document trends over time – if the student demographics change, if the population as a whole is getting better here, weaker there, so you can try and work out why.
Also, on the assessment side for measuring achievement – the problem with high-stakes testing isn’t that it’s a test (what else are you going to use?) it’s that it’s a measurement which has a variable you can’t account for – and that is time. Typically in assessment, you don’t care that they can all do it the very moment you ask them, so giving the student the test 10 times is perfectly fine, and giving them a different kind of test is fine so long as it measures the same thing. Once the student passes, you stop giving them that outcome measure and focus on the next one. These tests aren’t used for giving grades (that should include the behavior and other qualitative measures) just for determining if students are meeting the learning outcomes and for advancement.
Overall, it works pretty well if done thoughtfully. The problem with it is that it’s administratively expensive. If you don’t want teachers able to game the system (and they’ll do it subconsciously – I’ve only encountered one instance of it happening deliberately), you need to do it at the administrative level, and setting up and implementing a thorough assessment strategy takes a fair bit of time and labor, and these things take away from classroom time, but education administrators aren’t exactly held in high regard by the taxpayers.
Suffern ACE
O.K. but what is the outcome again once we know the measurement? Something is being measured here, but once we know that Teacher X is, say, 500 meters north of Brooklyn, and we want him to be 700 meters west of Hackensack, do we have any way to get teacher X to Hackensack? Springfield has a bad school because 75% of its teachers haven’t been rated well by their students. Now that we know who those teachers are, let’s….? Get rid of Ms. Krabappel?
Brian S (formerly Incertus)
@Mnemosyne: There’s a difference between catching a teacher in error and deciding that a teacher is doing a bad job, and I’m talking a lot more about the latter than the former. And I’m fine with putting together a system in which students can report teachers who are putting out demonstrably wrong information–it happens way more often than I’m comfortable with, especially in science and history classrooms.
But students don’t know what’s good for them most of the time, so to ask them to judge the quality of work that a teacher is doing isn’t fair to them or the teacher. It’s unfair to the teacher because it rewards teachers who are “cool” rather than competent. I’ll be the first to admit that I play my students when it comes time for evaluations, and I even let them know I’m doing it to a certain extent. But it’s also unfair to the students because it gives them a sense that they know something they really don’t, and that they’re qualified to make a judgment about a teacher without any sense of what’s required to run a classroom. It’s as if they’re being told they’re experts on a subject when they have no clue as to what’s required to do the job.
If there was one speck of evidence that student evaluations made better teachers, I’d be all for it. But I haven’t seen it, and I have no reason to believe it exists.
Brian S (formerly Incertus)
@Mnemosyne: Oh, about diagramming sentences–I’ll bet that you’ve put what you learned in that class to great use without even thinking about it. Don’t underestimate how few people actually know grammar and structure of language, even among English teachers. You don’t have to be able to diagram a sentence now to have gotten a fair amount of utility out of learning how to do it once.
debbie
Michelle Rhee’s started a national organization to accomplish what she tried to do in D.C.
http://www.studentsfirst.org/
sb
@debbie: Outside of massive layoffs and needless acrimony, remind again what she accomplished?
Davis X. Machina
@Brian S (formerly Incertus):
This keeps Latin teachers in business, feeds my children, pays my mortgage….
Brian S (formerly Incertus)
@Davis X. Machina: I really wish I’d taken Latin now, all these years later, and maybe one day I’ll find the time to at least get a primer course in it. My partner, Amy, took it as an undergrad and did some translation in grad school, and swears by it.
Brian S (formerly Incertus)
@Davis X. Machina: I really wish I’d taken Latin now, all these years later, and maybe one day I’ll find the time to at least get a primer course in it. My partner, Amy, took it as an undergrad and did some translation in grad school, and swears by it.
cthulhu
@Dervin: Or that particular cohort of students was unique in some other way (they formed study groups, encouraged each other, etc.). Sure, unlikely, but I’d be more convinced if this happened year after year for the same teacher (and cheating could be proven to not have happened). So really any honest assessment of a teacher will take several years unless they happen to be really horrible or really fantastic.
Suffern ACE
@sb: She became a superstar. Now where is she going to find all those great teachers?
Martin
@Brian S (formerly Incertus):
It doesn’t exist. Comments from students are helpful, because they’ll usually point out errors, and you do want some kind of feedback in that way, but you don’t want to base anything on that unless its systemic and can be backed up by empirical evidence. Otherwise, the student evaluations are useless.
Evaluations can be useful in a completely different regard, however. When coupled with effective feedback, it does help students feel more connected with the instructors and the school. But if that feedback doesn’t happen then it actually serves to reinforce notions that the students aren’t cared about. It can completely backfire, but if done right, it’s beneficial. Still not useful for actually evaluating instructor quality, though.
R. Johnston
Any value-added model for teachers that is a zero-sum model such as the given model is worse than useless. It’s a timebomb waiting to force teachers out on strike. A good teacher’s salary increase should never depend on some other teacher getting poor results. Any value-added model for teaching, in order to be useful, needs to measure value in a way that encourages teachers to support each other rather than encourages teachers to hope that other teachers fail.
Not to mention that the given model totally fails to distinguish between noise and actual value.
Martin
BTW, most of the problem with the education argument in the US is that it’s focused on ‘show me what this teacher has done’.
Who fucking cares? The point of education is not to extract a specific amount of work out of teachers, the point is to educate children. Until we learn how to shift the focus firmly there, rather than demanding that we get teachers justify their salary, then we’re never going to get anywhere.
Suffern ACE
@Martin: No. That can’t be right. If a school is full of underperforming students, it must mean that Robin Williams hasn’t come in to teach the kids the true meaning of life.
debbie
@sb:
Was it so needless? What was being changed so well before she came?
Ohio Mom
It’s hard to know where to start when listing all the things wrong with high-stakes testing. In no particular order:
HST eats into time better spent teaching. In our district, for example, starting at about six weeks before the testing week, all new learning grinds to a halt and it’s review and test prep, test prep, test prep. The half-week after the tests, it’s movie and popcorn time because everyone is totally spent. I’s a good chunk of the school year.
HST distorts and reduces the curriculum. What can’t be tested, gets ignored. Schools have cut out subjects that aren’t tested so they can drill the ones that are. And there’s less time for things like class discussion, group projects, and practicing public speaking (aka show-and-tell and oral reports) — things that can’t be tested (thank the FSM) but help develop important life/career skills.
HST is expensive! Those tests aren’t free. MILLIONS are being spent on them, and this in an age of reduced and strained budgets. And to what end? Do you think the teacher is surprised when Sally Smart does well and Spacey Sue doesn’t?
HST tests are just that, high-stakes. That means that the fourth grader who has a loose tooth that day, a tooth that demands it be wriggled back and forth with his tongue, all day long, taking his full attention from the task at hand — well, he screwed the test, and that’s it, there’s only one chance, that’s the definition of HIGH-Stakes.
Yes, there’s a place for testing in school. It can help a teacher determine a baseline, so she knows what the student needs to be taught, or needs extra help with. It can help um, inspire a student to work harder (I fully admit I probably wouln’t have learned a single irregular French verb without those weekly quizzes). But that’s not high-stakes testing.
As for this value-added stuff: if a student ONLY makes a year’s worth of progress in a year, what’s wrong with that?
sb
@debbie: Let’s change the question–how did she make it better? She was the focus of the mayoral race and voters couldn’t wait to get rid of her. Naturally, she was rewarded with millions of dollars to pursue her dream of killing unions and getting rid of any teacher who doesn’t measure up to her standards.
IMHO, Rhee is very good at self-promotion and not giving a shit who (other educators) or what (evidence contrary to her opinion) disagrees with her. As for what else she’s good for, I’m not sure. Frankly, I don’t see a lot of difference between Rhee and your typical wingnut.
buckyblue
Punchy: you stole my thunder. My best teachers were my hardest teachers that made me stretch further than I believed I could. I know that now, 30+ years later, but certainly not as an 18 year old senior. As a teacher, I know this, and I don’t have the adoring fans of some teachers, who never ask for homework, or a pass when they’re late for class, or anything that will demand their best. But, dammit, they didn’t hire me to be their friend, and if I was, you wouldn’t like me much anyway.
Tripmaster
@Martin
One other problem with the model you describe, aside from needing effective administrators, is the number of students that most teachers work with on a daily basis. Tracking 180 (or more) students becomes a bit problematic, especially with the rate of student movement between different different teachers within the same quarter, semester, or year.
MBL
I’ve got my ISTEP scores back for about 2/3 of the kids I taught English to last year. My sixth graders all passed. Each one of them. But they all passed in 5th grade too. My seventh graders I don’t know about yet. My eighth graders were interesting.
One kid who failed in previous years failed again. He’s ESL. He tries hard. But I knew he wasn’t going to pass.
Most of them passed in previous years and passed again; one went from a pass to a high-pass.
One kid who passed last year failed.
Four kids who failed last year passed.
Now, the individual scores don’t really matter; the only thing anyone cares about is whether they pass or fail, although the high-pass still looks pretty good.
Just by the numbers, I look pretty damn good, right? Pretty much upward movement everywhere. Okay.
The kid who passed last year and failed spent the entire year struggling with clinical depression and spent at least a couple of months of the year cutting herself for fun. So, her score? Not really my fault. Do I look even better now?
Of the four kids who went from passing to failing:
The first has passed every other year except last year, and missed a pass last year by ONE point.
The second’s father shot himself in the head two days before last year’s ISTEP test. Her test counted anyway.
The third is one of the best middle-school writers I’ve ever encountered, and I have no idea how the hell she failed language arts last year. I don’t know what she did in fifth & sixth grade, but I’m considering seventh grade an aberration.
The fourth I think genuinely did better.
So: Those are my test scores.
Am I good at my job?
terry chay
It’s a loser that and defies all common sense.
The implication of all value-add research is that you can now pay teachers their “value” As if the cause of all that ails our educational system can somehow be pinned on unionized and overpaid educators. No study ever has proved THAT canard.
If people went into teaching because of profit motives then they are stupid as rocks. A zillion studies have shown in fields that require creativity at all, pay for performance is the worst incentive model out there. Common sense will tell you what tying a teacher’s pay to a “value add” model will do to cohesion in a typical high school—it’s not good.
The net result, when the studies come back after a few years down this rabit hole—irrespective of the accuracy of the “value add” model—the application of it will be to do far more harm the schools far more than even a modest improvement they offer.
And then we’ll have another stupid “Waiting for Superman” documentary on another poorly-applied social experiment that even a minimally educated person with a modest amount of common sense could have told you was an emminently dumb idea.
That people believe this or charter schools actually work and mount a ton of evidence to prove what is obviously not true as being true, and even well-educated liberals make documentaries about it—well that is proof at just how bad our education system truly is—that it cannot produce a single person with common sense.
Morat20
Bear in mind: Your crappy sixth grade teacher you’d downrank because she felt that all you needed to do in English that year was diagram sentences?
Yeah, she didn’t think that. Your state school board had a bug up it’s ass over falling English scores, and had three new locally elected idiots running in a race no one ever pays attention to who ran on “returning to the Three Rs” and managed to get enough votes to force a change in curriculm mandating an outdated and particularly painful rote technique for learning to write — sentence diagramming. OVER AND OVER.
Your teacher? Probably hated it. Or perhaps it wasn’t the state, but your local school board.
See, teachers aren’t tin-plated Gods in their classroom. They’re given a strict curriculm from the state — generally built entirely by non-educators — and then another from the local school board, and then have to cater to whatever pet ‘technique’ the new superintendent has to push scores higher.
Or heck, maybe she loved the sentence diagramming. But you, as a 6th grade student, are utterly unaware of whether or not she liked, loathed, or tolerated sentence diagramming. Because to you, as a sixth grade student, the teacher was the boss of the classroom, and she did whatever the heck she felt like.
Truly excellent teachers, if they have a fully supportive administration, can and do subvert the most idiotic and damaging dictates from local and state school boards. Sadly, excellent teachers are thin on the ground in such a low paying field, and supportive administrators….they’re even more rare.
And after 20 years of fighting the good fight, somehow squeezing in actual education around whatever moron crap you’re made to focus on for low pay and constant disrespect, you burn out.
Because always, in the back of your mind, is the mantra: “Teaching’s not a real job. It’s what people do when they can’t get work in their real field. It’s easy. Anyone can do it. It’s women’s workso how hard can it be?”. And you grit your teeth, and snarl, and wonder how many people blithly telling you how overpaid you are and how you get the whole summer off realize you have a double Master’s degree, spend every summer doing professional development to stay current and good, make 35k a year, work 70 hours a week during the school year (teaching + prep + grading + required “volunteer” stuff) all for the joy of listening to some dentist on the state school board tell you how to do your job, or listen to Sally’s parents explaining how you were wrong to give her a detention for failing to turn in her work for a month, or perhaps for trying to stab another student with a pair of safety scissors…..
Oh, and I forgot: People who’d vomit if forced to actually face a classroom, trying to measure your performance, and using tools that indicate they think you get to toss back the defective kids.
Suffern ACE
@terry chay: This.
@Morat20: This.
@MBL: Keep up the good work.
Suffern ACE
@terry chay:
That’s really a function of our higher education system. Reforming the fact that it spews out a crummy elite is a topic worthy of discussion on its own thread. Instead, they kind of like to blame the teacher’s union when the kids aren’t “prepared to compete in the 21st century” when no amount of education reform is going to produce middle class jobs for everyone. We don’t have jobs for you because your teachers were lousy…maybe there is another reason for that.
Martin
@MBL:
Yes, because what we care about is the kids passing (or more specifically that they are all on upward trajectories), which they did. We only care about what you are doing if they don’t.
The anecdotes you describe happen everywhere and they average out in the data. While they may have had a pile of issues outside of your control, the fact is that you still needed to deal with those issues, and you may have helped bring some of those items to light. What didn’t happen is that the other students do more poorly because of that.
Bottom line, the students passed and did better. Since that’s all we should care about, yeah, you should be rewarded. The details don’t matter.
MaximusNYC
It seems to me that the value-added system has merit. Of course it can’t tell you much about an individual student. It may not even tell you much about an entire class. But looking at relative performance of different teachers’ classes over years, you should be able to see important trends, especially when the teachers are teaching the same subjects to the same types of kids.
At the very least, it should be useful as a warning sign of an especially poor teacher.
I think student evaluations, if well-conceived, can be useful too. As they say at Slashdot, RTFA (Read the F*ing Article): It makes clear that these are very carefully thought-out questions, that ask about whether the teacher manages the class well, keeps students learning, etc. It’s not a popularity contest.
As a public school student, I experienced a wide spectrum of excellent and terrible teachers. I could have told the school board about some of the egregiously bad ones, including a verbally and physically abusive woman I had in 5th grade, who, I recently discovered, is still teaching almost 30 years later! She’s also a prominent union rep. Call me cynical, but I see a connection. (And I’m very much pro-union in general.) I found her taking potshots at Arne Duncan in the comments on his Facebook page, over this very issue. It just made me believe even more strongly that value-added evaluation is a good idea.
My parents did complain about this teacher, but nothing happened (other than me being transferred out of her class). I do believe there is a need for objective criteria that can be used to get the very small number of truly bad apples out of the system.
Menzies
@burnspbesq:
Sorry it took me so long to reply to this.
I have a real problem with value-added modelling because it relies (to my understanding) on standardized testing, and in the first place I have an issue with that. In my cynical, reductionist view it’s exactly the kind of policy a former basketball player (that is, a stat-tracked athlete) would want to implement – a strict numerical ranking – but obviously Duncan isn’t alone in pushing it and it’s not just his view that makes this kind of stat sexy.
I agree with what you say about classroom observation, which is why I think it’s valuable to ask the kids, and it’s not like I think teachers should be immune to evaluation by the people they work with and the people who study under them. Problem is I also think when you tell people “this is how many points Mr./Ms. Whatever adds, on average, to a kid’s exam scores,” you’re only helping the simplification of all of the things that go into good teaching to a number. It’s not parents or administrators being stupid, it’s they want magic bullets. I know you said that it should be only one of various tools, but I find it particularly valueless among the many that can be used.
tgp
Standardized tests have been in effect for decades at this point. im sure its possible to track students from 1st grade through high school. if monitoring the changes in year to year performance of students is a worth while thing it should be very easy to find the good performers from the bad in the data.
i suspect that the results of looking at the year to year trends on scores really isnt all that clear. lots of noise and no clear causal variable for any of the fluctuations in scores.
before i started relying on this for much i would need to see real trends over substantial amounts of time. if you show me that over 10 years a teacher’s students do better relative to their peers then that might be useful to know. year to year changes seem to open to random fluctuations and all the confounding variables. additionally you would have to take into account the fact that students could overperform or underperform from year to year totally unrelated to the teacher. so you would need to construct baselines for that student. this would require a several year sample size so that really the model isnt useful until 5th or 6th grade and only for teachers with 5 yrs experience.
eyelessgame
I can’t possibly be the only one who’s noticed that this is a zero-sum game, and that no matter how hard the teachers work the net amount of value they add will be zero.
Morat20
Offhand, with VAT, no one’s going to want to teach the 6th through 9th grade classes.
Have you ever met pre-teens? Fluctuating hormones and the associated pouting, moping, ‘dating’ associated drama (I use ‘dating’ in quotes because junior high dating is something all on it’s own), and the dawning “hey, wait, what happens if I don’t do this?” ability to question authority and test boundaries…
Sadly, I’ve been told I can’t send my 14 year old back to the factory, and that his current behavior is per design. Not that there aren’t consolations.
Watching him fluctuate between a sullen teenage pout and a full-scale 11 year-old tantrum over the fact that he got grounded for mysteriously failing to turn in a raft of English assignments was, at times, hilarious. His voice cracking during his ‘defense’ was even better.
It’s strange. It’s like he rotates which class he simply doesn’t do the work in. “Oh, you got straight A’s except for a low B in English. Which was the result of not turning in four assignments.” turns into, next six weeks, “You got straight As except for a low B in geometry because, and this sounds familiar, you didn’t turn in all your work“.
It’s not like he’s overly burdened with homework. He just seems to pick a class and not do it. It’s driving me nuts, but I’ve been informed by a number of professionals that it’s part and parcel of the developmental process, and given some ways to help keep him on track that work better than throwing a massive “WHY WON’T YOU DO YOUR HOMEWORK? FOR GOD’S SAKE!” tantrum followed by massive punishment.
Not that getting his little butt grounded isn’t part of it, but apparently — and this is shocking me — understanding the weird little crazy that is the 14 year old brain actually helps.
jcgrim
One of the overall problems with this the focus on value-added assessment is it is based on the false assumption that public schools are failing due to poor teachers and administrators. Years of research demonstrates poverty is the primary predictor of school failure, and trumps every other indicator.
Using unreliable and invalid measures is not a concern for the edu-reformers. This push away from reforms teachers have been demanding for 35 years (small class sizes, broadening curricular choices, increasing in-classroom resources) are ignored for ideological ends. Ends which are a rabid free-market approach to education.
Don’t think for one minute Arne Duncan’s reforms, accountability testing, and Race to the Top’s mandate to double the number of charter schools is about improving poor schools. It’s about transferring tax dollars to private investors under the guise of “getting rid of poor teachers” or “saving inner city children”.
They don’t care if kids are over tested, if the tests are unreliable indicators of instructional quality, or if charter outcomes are no better than public schools (all facts).
The motivation is shareholder profits.
Corporations are investing in charter schools and using their public “mandate” to:
invest in real estate: http://www.stickwithanose.com/…
getting tax breaks that double their investment:
http://www.democracynow.org/2010/5/7/juan_gonzalez_big_banks_making_a
jcgrim
Do you trust this magnanimous group to reform schools?
http://us2.campaign-archive.com/?u=cfb5b3dae0d0b7da2e949d8d1&id=078f7efb22&e=2e4c6c94e1
http://us2.campaign-archive.com/?u=cfb5b3dae0d0b7da2e949d8d1&id=078f7efb22&e=2e4c6c94e1
Cheryl from Maryland
Having no math skills or children, I can only offer anecdote. On the Top Chef challenges involving children, cheftestants that appeal to the stereotype of kid’s food loose. Those that cook creative food, just not as heavy on the spices, win. And even though polled separately, the children and the judges, especially Tom Colichio, agree. Thus, interviewing the students may be useful as children aren’t the self centered thoughtless selfish beasts many assume. Once they are teens though …
Dervin
@Brian S (formerly Incertus): @cthulhu:
Really is that the best the both of you can come up with? Forget about the baby Jesus crying, you guys just aborted logic and rationality. When we fly, we have to take our shoes off, dispose of all our water, leave our nail clippers at home because of the irrational paranoid imagination of people like you.
Your fever dreams would lead to us getting rid of traffic lights because people run red lights.
This is a nice tool which can benefit, students, parents, teachers, administrators but out of the millions of teachers (6.2 million teachers) there will be two dozen teachers who’ll try to cheat the system.
Exurban Mom
My experience as a parent and heavily involved volunteer in the public schools leads me to the following conclusion:
If we really want good teacher evaluation, so that we can get rid of underperforming teachers, we need an “all of the above” type approach. We need student evaluations, PARENT evaluations (of things like communication to the home, child’s experience in the class as reported to us), administrative evaluations, peer evaluations, and test scores.
The person above (lost track) who claims that anyone who says their job is hard to evaluate is trying to snow you? In this instance, it IS difficult. It requires a comprehensive approach. I agree that students are usually thoughtful critics, but even the smartest, most reasonable students can get caught up in a feeding frenzy against a teacher who does something the kids don’t like. I’ve seen it happen.
And the other thing: this type of Value-Added system disadvantages those who teach children with disabilities (or gifted kids, for that matter), and discourages teachers from having these students in their classrooms.
If a gifted kid is very high percentile-wise, it will take a good amount of effort to get that kid even higher for the next go-round of testing, perhaps pushing the teacher outside of the curriculum he/she uses for the rest of the class. Kids with learning disabilities require an intense amount of effort to get them on grade level and progressing. This type of testing disincentivizes teachers from working with these kids. I’m not saying teachers are bad people–I’m saying that if your job DEPENDED on you achieving certain results with test scores, these types of students might not receive the proper attention and guidance.
Better supervision by better administrators would help to deal with a lot of teacher problems, but often, our administrators are either overburdened with dealing with other problems, or not super interested in monitoring classroom performance.
Shinobi
I know I am super late on this, but, I have to say this, unequivocally:
Any attempt to evaluate a complex system using a single score will eventually be shown to be ineffective.
This was true of the risk scores they used to evaluate home loan packages before the recession, it is true of Net Promototer Score, and NPV. One Number is simply not enough information to make a well informed evaluation on most complicated situations.
I would also be really interested in seeing how accurate their forecasts of future scores are to begin with. That is a major chunk of this analysis, and forecasting is a difficult and deeply imprecise art. If their estimates of future scores were wrong, then their evaluation of the teacher would be wrong.
Also this problem is make even more problematic by small class sizes. The smaller the class size the more unstable the estimations would become. If one student had a bad cold the day of the test it could send the teacher’s scores into a lurch. Conversely if a bad teacher just happened to have a student who had done a lot of learning on their own that year, it could make that teacher look like a better teacher than they are.
As a statistician I think this could be an interesting theoretical model of learning and teaching behaviors. As someone who works in the real world and sees how management tools are employed, I cannot say NO loud enough.