The Future of Empirical Legal Studies

Kenesaw Mountain Landis would have hated both sabermetrics and ELS.

Reading these two articles on the problems of complexity for sabermetrics, I wondered if the empirical legal studies community is coming soon to a similar point of crisis. The basic concern is that sabermetricians are devoting oodles of time to ever-more-complex formulae which add only a small amount of predictive power, but which make the discipline more remote from lay understanding, and thus less practically useful.   Basically: the jargonification of a field.  Substituting “law” into Graham MacAree‘s article on the failings of sabermetrics, we get the following dire warning:

“Proper [empirical legal analysis] is something that has to come from the top down ([law]-driven) rather than the bottom up (mathematics/data driven), and to lose sight of that causes a whole host of issues that are plaguing the field at present. Every single formula must be explainable without recourse to using ridiculous numbers. Every analystmust be open to thinking about the [law] in new ways. Every number, every graph in a [ELS] piece musttell a [legal] story*, because otherwise we’re no longer writing about the [legal system ] but indulging in blind number-crunching for its own sake. …

Surveying the field, I no longer believe that those essential precepts hold sway over the [ELS] community. Data analysis methods are being misapplied and sold to readers as the next big thing. Articles are being written for the sake of sharing irrelevant changes in irrelevant metrics. Certain personalities are so revered that their word is taken as gospel when fighting dogma was what brought them the respect they’re now given in the first place. [ELS] is in a sorry state.

How do we fix it? Well, the answer seems simple. [ELS] shouldn’t be so incomprehensible so as not to call up the smell of [a courtroom, or the careful drafting of the definition clauses in a contract, or the delicate tradeoffs involved in family court practice, or the importance of situation sense]. Statistics shouldn’t be sterile and clean and shiny and soulless. They shouldn’t just be about [Law]; they should invoke it. Otherwise, they run the risk of losing the language which makes them so special.”

Note: this is an entirely different  than Leiter’s 2010 odd critique that ELS work was largely mediocre.  The problem, rather, is that the trend is toward a focus on more complex and “accurate” models, often without the input of people with legal training, and insufficient attention to how such models will be explained to lawyers, judges and legal policymakers.  (See also all of Lee Epstein’s work.)

You may also like...

3 Responses

  1. Matthew Lister says:

    Do you mean to be identifying Epstein’s work with “the problem” you identify in the last paragraph, or as something pointing out this sort of problem? I was unclear from how you’ve put it.

  2. Dave Hoffman says:

    Lee’s work says what this blog post did, and offers solutions. Sorry if that was unclear!

  3. Matthew Lister says:

    Thanks Dave- I’ve only read a small portion of Epstein’s work, but didn’t see it as being problematic in the way the post suggested some empirical work is, so wanted to make sure of what you were saying.