the perils of data

When this article ran in the NYT magazine, several months ago, I had a whole post planned out about one particular thread. Joe Nocera describes the evolution of Value at Risk - VaR - which was a system JPMorgan developed for measuring risk. It became the financial industry standard for measuring risk for a number of reasons: it gave a single number for the primary riskiness, JPMorgan developed it and then gave it away, and it gave bank regulators a simple thing to look at to figure out if banks were taking on too much risk.

Nassim Nicholas Taleb points out that there's a whole set of events out beyond the 99% of normal variation that VaR covered which, over time, became very significant; there were also several critiques of Nocera's article which pointed out that VaR assumed that prices essentially varied randomly, and couldn't account for real-world events that affected risk. I've lost the links to those articles, or I'd link to them - they were by actual economists who actually know things.

But here's what's more interesting to me than the near-collapse of the financial system: it's a problem that Nocera does cover, by summarizing what Till Guldimann, a former JPMorgan banker involved in creating VaR, told him:

"The big problem was that it turned out that VaR could be gamed. That is what happened when banks began reporting their VaRs. To motivate managers, the banks began to compensate them not just for making big profits but also for making profits with low risks. That sounds good in principle, but managers began to manipulate the VaR by loading up on what Guldimann calls 'asymmetric risk positions.' These are products or contracts that, in general, generate small gains and very rarely have losses. But when they do have losses, they are huge. These positions made a manager’s VaR look good because VaR ignored the slim likelihood of giant losses, which could only come about in the event of a true catastrophe."

In other words, the people who created the policy environment built incentive structures around a particular data point. So the people operating in the policy environment privileged that data point over all others. Turns out that credit default swaps look very good in a VaR model; turns out they also create huge systemic risks by entangling many financial actors into any particular problem.

Can anyone think of any other time this has happened? Like maybe in higher education, with a set of rankings? Or how about in K-12 education? Oh that's right, it's called "accountability." It's what NCLB would be doing, if it had more teeth.

We live in an age where people are very interested in data, and in a lot of ways that's great. We should try to figure out how to measure things: the same NYT article mentions a situation in which a recurring data point tripped some managers' attention at Goldman Sachs, and as a result they met, discussed the mortgage market, and decided to take on less risk. That's a good use of data. But blindly privileging any particular data point will leave any system vulnerable to being gamed. I guarantee you that there are schools out there that are figuring out how to game - not cheat, but game - the testing system. Some of those schools are also doing a good job on other things; others are focusing on specific tests, at a real cost to their students. My school tried to game the test by setting up a special academy for students they thought might make 'proficient,' and having higher behavior and academic standards for that academy. It may or may not have helped those students; it certainly didn't help anyone else.

The same thing is happening with Clemson University in the Inside Higher Ed article: some of the reforms they're making are good for their students, others are attempts to game the system, but none of them proceed from an honest evaluation of what would make Clemson a better university. It's schmality instead of quality, and I wish the data evangelists would be honest about the way a laser-like focus on data makes the pursuit of schmality worse.

something short about education

The Economix blog at the NYT recently posted, describing the comparative inefficiency of the US educational system. Which, if you look at the graph, is true. But it's more complicated than that. (I posted a chunk of this as a comment on that blog.)

Two major issues: spending per student in the US is in fact very split between wealthy and poor districts. Philadelphia spends $11,000 per student per year, almost $10,000 less than nearby Lower Merion, and is constantly short of funds. Not that there isn’t waste in Philly’s system, but money certainly isn’t easy to come by for teachers.

Second, the other countries which do well tend to have strong, generous social support networks. My guess (as a former teacher) is that schools there don’t have to provide health care, counseling, food, etc to students who have trouble getting them at home. One huge difference between my experience going to school and my experience teaching was that, by and large, the kids I went to school with got glasses when they needed them, and if they fell way behind in reading or math, their parents noticed (were equipped to notice by their own educational success) and got them tutoring or other assistance (because they had either time or money with which to provide those things).

There are still huge problems within our educational system that are matters of educational policy rather than social policy more broadly defined: we lack a unified set of national standards, the standards we do have (especially in math, though this may also be true in other subjects where I know the debates less well) are less rigorous and more connected to rote memorization than the standards in other countries, and teachers have far less prep and development time than teachers in, for example, Japan.

I’m routinely astonished, though, by the number of non-educators (and sometimes educators) who think that failing students is somehow the key. This always, always comes up in comments on articles about education, and usually some version of this will also come up in a professional development workshop. Yes, students need accountability, and yes, it makes your job a lot harder when students don't know the earlier material. But making them repeat the same material in the same context without additional supports doesn’t help - it just leads to a bored, frustrated kid who thinks he or she has already learned this (and has, in the sense of having sat in a classroom while it was being taught). The evidence also just doesn't support a claim that making students repeat a grade improves their achievement - mostly it makes them more likely to drop out, and lowers over-all achievement. You can’t just keep doing the same thing and expect different results.

so classic! and so wrong!

Most important, it would increase merit pay for good teachers (the ones who develop emotional bonds with students) and dismiss bad teachers (the ones who treat students like cattle to be processed).

There's so much wrong with this as a statement of policy that I don't even know where to start. You know who wrote it, right? David Brooks. Who seems to believe that emotional bonds with students are both the real measure of success in teaching, and that they can easily be measured using standardized tests. Honestly, there are a lot of people who believe both of these things. They're all wrong.

Meaningful, personal relationships with students are great. They were and are the lifeblood of any success I ever had as a teacher, in no small part because they were by far the most rewarding part of teaching and I would never have lasted without them. I still run into my students occasionally - on the trolley, on Facebook, and outside the deli by my house - and the ones I see, for some reason, are students I had a real relationship with. Every so often I get a phone call or an email. I love knowing how my students are, and sometimes they tell me that my role meant something. But - and I know this is true - this is only one of the many ways to be a good teacher, and it is not enough. I knew plenty of teachers who had real relationships with their students, who did not treat them like cattle to be processed, and who nevertheless did not expect their students to do well academically and were unable to get them to do so. In fact, there were plenty of my students - the students of someone who unambiguously cared about her students, and tried to have meaningful relationships with them - who for various reasons, didn't learn that much in my class. For some of those kids, I know why: untreated mental illness, chaotic life situations, illiteracy, a rational calculation that summer school would be less work. For others, I don't. There were kids with whom I had real connections who didn't learn much.

While I didn't have kids who couldn't connect, but nevertheless learned, those kids - and those teachers - exist. It's a different teaching style, and while it's unlikely to help a kid supersede crazy obstacles outside of school, that impersonal quality can be its own powerful center for a classroom culture. I had plenty of teachers like that in high school, and some of them were excellent: they devoted tremendous attention to planning their lessons, communicating material, and offering academic feedback, with minimal interest in your personal affairs. In some situations, this is a great teaching method, though I'm not surprised that David Brooks, decades from any personal experience as a teacher or student, can't remember the value of these teachers. I will agree that many students need to have a few teachers who develop a personal connection and use that to motivate that individual student, and that students from unstable home situations can especially benefit from that kind of mentoring, but it is nevertheless not the only valuable teaching method.

Brooks's worst mistake is to claim that merit pay will reward caring teachers over impersonal ones. The obstacles to implementing merit pay are enormous: most systems give good teachers even more incentive to find a well-run, high-performing school serving students with stable home resources, and even less incentive to work with the most difficult students. At any given school, the obsessive focus on standardized test scores takes time: teachers who are interested in how their students will do on standardized tests, and thus what their merit pay will look like, need to teach lessons focused on standardized tests, research the standards, grade practice tests, etc. None of those things involve real connection with students. And real connection with students will continue to go unrewarded, because it is remarkably difficult to measure, and in and of itself not sufficient.

It's no surprise that David Brooks is incoherent, and the rest of the article (where he tries to talk about policy) is worse. But this tiny example - one sentence in one column - seems extraordinarily apt to me as a representation of just how poorly thought out his views on education are.

(I should probably leave this alone, but I just can't:

Democrats in Congress just killed an experiment that gives 1,700 poor Washington kids school vouchers. They even refused to grandfather in the kids already in the program, so those children will be ripped away from their mentors and friends. The idea was to cause maximum suffering, and 58 Senators voted for it.

There is practically no evidence that vouchers work. They do not provide adequate funding for most students to attend wealthy, fancy private schools, and they disproportionately benefit students with well-organized, stable family situations who are in the best position to take advantage of it. Plus, vouchers are expensive. I'm not deeply opposed to grandfathering in students who already have vouchers, but it's fucking irresponsible for David Brooks to talk about it this way. Education is complicated, and there's a lot of real information out there. He needs to shut up til he understands it.)