What Should a Judge’s Reversal Rate Be?
Via TPM, I found that critics of Judge Sotomayor have made some hay of her reversal rate in the Supreme Court (50% of 6 cases heard). Though I like data more than the next guy, I will freely admit that this use of quantitative empirical legal studies is demonstrably silly. I’m going to try to demonstrate to you in this post that a reversal rate of fifty percent is exactly what the state-of-science would predict to be ordinary [or even below average], and a rate of zero (which some might think of as emblematic of a law-abiding judge) tells you that the judge in question might be departing quite severely from precedent.
Let’s start with the obvious. Most appellate court opinions aren’t subject to a cert petition; almost no cert petitions are granted; only some granted cert petitions end up in full argument and a written opinion by the Supreme Court. Throughout this long process, parties may settle their cases and exit the system. They may do soduring briefing before the appellate court or after argument (with some limitations), they may settle after an opinion issues but before a cert petitition is filed, and they may settle thereafter until the Court rules (again, with some limitations based on mootness doctrine). This potential for settlement after the appellate court issues its mandate creates selection effects.
Though such selection effects are likely less predictable & more dominated by wealth & party characteristics than the immense selection that occurs in the district court, it remains that case that the universe of cases that survive the decision to appeal and the decision to grant certiorari is significantly winnowed. That winnowing produces a distinct set of cases. Cases before the Supreme Court contain legal & factual issues more finely balanced than those that issued from the courts below. To put it another way, cases are argued (usually) because the parties both believe they are going to win. If the parties are rational & wealth neutral, an assumption that sometimes holds, we should expect that the resulting decisions from the Court will be a bit of a random walk. (See my earlier post on bankruptcy scholarship for more on this hobby-horse of mine. Also, note that much of this applies to civil cases, not criminal cases, which create unique settlement patterns. The actual rate of reversal, over all cases, ranges between 60 and 75%.)
What’s the upshot? An appellate judge’s “reversal statistic” tells you less than you think about the “merits” of her opinions, or even how such opinions stacked up against governing Supreme Court precedent. Ironically, when a judge significantly departs from precedent either for or against the plaintiff, settlement may be particularly likely, as the parties’ chances above are quite clear: thus a judge who convinces her colleagues to depart from precedent often will almost never be reviewed or reversed by the Supreme Court. Cases where the judge stuck with precedent, by contrast, may face appeal and reversal, especially if the precedent shifted due to ideological change on the Court itself.
There’s lots of good work on this, much of it recent. And what it teaches me is that we have no idea what an appropriate reversal rate for an appellate judge ought to be. It also suggests that there is no way to evaluate the quality of an appellate judge’s work except to read her opinions and decide for yourself what you think of them. This is a clear instance where statistics mislead.