What Should a Judge’s Reversal Rate Be?

Via TPM, I found that critics of Judge Sotomayor have made some hay of her reversal rate in the Supreme Court (50% of 6 cases heard).   Though I like data more than the next guy, I will freely admit that this use of quantitative empirical legal studies is demonstrably silly.  I’m going to try to demonstrate to you in this post that a reversal rate of fifty percent is exactly what the state-of-science would predict to be ordinary [or even below average], and a rate of zero (which some might think of as emblematic of a law-abiding judge) tells you that the judge in question might be departing quite severely from precedent.

Let’s start with the obvious.  Most appellate court opinions aren’t subject to a cert petition; almost no cert petitions are granted; only some granted cert petitions end up in full argument and a written opinion by the Supreme Court.  Throughout this long process, parties may settle their cases and exit the system.  They may do soduring briefing before the appellate court or after argument (with some limitations), they may settle after an opinion issues but before a cert petitition is filed, and they may settle thereafter until the Court rules (again, with some limitations based on mootness doctrine).  This potential for settlement after the appellate court issues its mandate creates selection effects.

Though such selection effects are likely less predictable & more dominated by wealth & party characteristics than the immense selection that occurs in the district court, it remains that case that the universe of cases that survive the decision to appeal and the decision to grant certiorari is significantly winnowed.  That winnowing produces a distinct set of cases.   Cases before the Supreme Court contain legal & factual issues more finely balanced than those that issued from the courts below.  To put it another way, cases are argued (usually) because the parties both believe they are going to win.  If the parties are rational & wealth neutral, an assumption that sometimes holds, we should expect that the resulting decisions from the Court will be a bit of a random walk.  (See my earlier post on bankruptcy scholarship for more on this hobby-horse of mine.  Also, note that much of this applies to civil cases, not criminal cases, which create unique settlement patterns.  The actual rate of reversal, over all cases, ranges between 60 and 75%.)

What’s the upshot?  An appellate judge’s “reversal statistic” tells you less than you think about the “merits” of her opinions, or even how such opinions stacked up against governing Supreme Court precedent.  Ironically, when a judge significantly departs from precedent either for or against the plaintiff, settlement may be particularly likely, as the parties’ chances above are quite clear: thus a judge who convinces her colleagues to depart from precedent often will almost never be reviewed or reversed by the Supreme Court. Cases where the judge stuck with precedent, by contrast, may face appeal and reversal, especially if the precedent shifted due to ideological change on the Court itself.

There’s lots of good work on this, much of it recent.  And what it teaches me is that we have no idea what an appropriate reversal rate for an appellate judge ought to be.  It also suggests that there is no way to evaluate the quality of an appellate judge’s work except to read her opinions and decide for yourself what you think of them.  This is a clear instance where statistics mislead.

You may also like...

7 Responses

  1. Steven Lubet says:

    Doesn’t SCOTUS reverse slightly more than 60% of all cases?

  2. Dave Hoffman says:

    Steve, I think that’s a blended crim/civ statistic. But you are of course right that the overall reversal rate also strongly suggests that selection & case selection by the justices is operating pretty strongly.

  3. Howard Wasserman says:

    There also is the simple fact that being reversed does not tell us anything about whether the lower-court judge was right or wrong, only that her views were or were not in step with those of the reviewing court. We are back to the old saw: We are not final because we are infallible, we are infallible because we are final.

  4. Jake says:

    Does it make sense to evaluate a federal circuit court judge’s reversal/affirmance rate by limiting the analysis to cases that are granted cert based on circuit splits?

  5. Sam Baumgartner says:

    Apart from selection effects, a “sample” of six is not exactly large enough to draw valid descriptive inference, in this case the implied inference from her reversal rate to her level of fidelity to precedent.

  6. Glenn Johnston says:

    Please review the following stats on reversals:
    2009*: http://www.scotusblog.com/wp-content/uploads/2010/07/Final-Charts-070710-10.pdf

    Partial 2008 Stats: http://www.scotusblog.com/wp-content/uploads/2010/06/Preliminary-Stats-OT09_062610-8.pdf

    All else available on Scotusblog.

  7. Michal Zapendowski says:

    The Supreme Court only agrees to review about one percent of the cases submitted to it. That is what accounts for the high reversal rate.

    A large proportion of the cases the Supreme Court hears are cases that, for one reason or another, are of such great importance that the Court has little choice but to hear them. Obamacare was an example — several U.S. Courts of Appeals had declared the federal law unconstitutional, several others had disagreed, and if the Supreme Court didn’t step in to clean up the mess, it would have wreaked havoc on implementation.

    What that means is the one percent statistic is really an overstatement. For cases on which the Supreme Court really has a choice on whether to hear the case, the percentage accepted is lower than one percent.

    So within that narrow window of discretionary cases, which ones do you, as a Supreme Court Justice, want to address — cases whose result you agree with, or ones that you believe to be a violation of the Constitution or a miscarriage of justice, and want to correct?

    If the Justices agree with a lower court’s decision, they can simply refuse to hear the case and thereby passively “affirm” the lower court’s decision. This doesn’t have the legal effect of affirming on paper, but it has that practical effect for the parties in the case, and also for the courts that fall within the lower court’s appellate jurisdiction.

    This is why the Supreme Court has a reversal rate of over sixty percent, whereas lower courts of appeals (which address every appeal filed) have substantially lower reversal rates.

    So, first of all, Justice Sotomayor’s reversal rate of 50% is actually low.

    Secondly, the reasons that the Supreme Court reverses are also different from lower courts. Typically, a Supreme Court opinion will split on ideological lines. The Court will often take on a case not in order to “correctly apply the law,” but in order to change it. Lower appellate courts do this as well, but far less frequently. The vast majority of reversals in the lower courts of appeals result from errors made by the trial courts.

    So the fact that Justice Sotomayor was reversed 50% of the time in the Supreme Court only shows that, for whatever reasons (and reasons likely to be ideological), a majority on the Supreme Court disagreed with her 50% of the time. It doesn’t reveal whether she followed precedent, or didn’t follow precedent. It doesn’t reveal whether she “correctly” applied the law in her cases. The fact of the matter is that the Supreme Court doesn’t police these things. It renders ideological decisions in order to shape the law, and for the Justices on the Court, the “correct” application of the law is “whatever we say it is.”

    So in summary, if Sotomayor were a trial judge and had a reversal rate of 50% in an ordinary court of appeals, that would reveal that she either did not follow precedent, or made a lot of errors. But a similar statistic from the Supreme Court means neither of these things, because the Supreme Court is very different from an ordinary court of appeals, and reverses for entirely different reasons.