Why is Bob Morse, USWR’s ranking guru, so unafraid of competition? There’s plenty of evidence that his rankings are fragile – for example, changes at the “bottom” of the scale have unexpected influences on schools at the top because the magazine engages in a statistically bizarre forced normalization – and that they measure factors that have little relevance to an underlying “quality” measure of importance in the market for legal jobs. But he persists in acting like an incumbent politician, prioritizing optics over real reform.
Here’s an example. A few weeks back, Bob Morse issued a stern warning to law school administrators out to game his rankings. In response to a problem created by “openness about our ranking model” Morse took a strong step in the direction of reform by…wait for it…threatening certain schools with punishment for gaming their employed-at-graduation statistic. For those who follow the rankings, this was a particularly galling and obnoxious post. The rankings model isn’t at all “open”: for most categories of concern, USNews engages in hidden manipulations of dubious value which make replicating the results quite difficult. See, e.g., LSAT percentile scoring, COLA adjustments; normalization, treatment of missing data, etc. Indeed, the rankings would likely fail the very low bar for openness and replication set by even a student-edited law journal, let alone a peer reviewed publication. And the irony is that savvy administrators would find the at-employment graduation statistic of very little interest, because its relative contribution to a school’s rank is significantly dampened by lack of variance. To put it another way, Morse seized on “gaming” of a factor that has the second least connection with overall ranking success (behind library volumes). Leiter said it best: “fiddling while rome burns.”
Why did Morse’s team focus on this particular statistic, as opposed to working on real reform? It’s all about the perception of legitimacy: an increasing number of schools weren’t reporting their employment numbers (because the formula for imputing missing values in this input factor produced a helpful number). When Paul Caron pointed this out, it was embarrassing for Morse, especially when Caron’s post was picked up widely. Real reform might result in dramatic changes that called into question earlier rankings, as well as adding cost and expense. This change, on the other hand, will have at most a marginal effects on scores, and costs essentially nothing. Mission accomplished!
Still, the question remains unanswered: what makes Bob Morse so convinced that incumbency renders his flawed product insensitive to normal market corrections?