A common criticism one reads of ELS is that “too much of the work is driven by the existence of a data set, rather than an intellectual or analytical point.” It’s ironic that this is the very critique that the realists made of traditional legal scholarship. Consider the great Llewellyn:
“I am a prey, as is every man who tries to work with law, to the apperceptive mass. I see best what I have learned to see. I am a prey, too — as are the others — to the old truth that the available limits vision, the available bulks as if it were the whole. What records have I of the work of magistrates? How shall I get them? Are there any? And if there are, must I search them out myself? But the appellate courts make access to their work convenient. They issue reports, printed, bound, to be had all gathered for me in the libraries. The convenient source of information lures. Men work with it, first, because it is there; and because they have worked with it, men build it into ideology. The ideology grows and spreads and gains acceptance, acquires a force and an existence of its own, becomes a thing to conjure with: the rules and concepts of the courts of last resort.”
Or to put it differently, all of our work – quantitative empiricists, doctrinalists, corporate finance wizards, administrative regulation parsers, legal philosophers, and derivative social psychologists alike – is driven by the materials at hand. For most lawyers and legal academics, appellate opinions are the most convenient pieces of information available; we use such opinions to create mental models of what the “law” is, and (ordinarily in legal scholarship) what it ought be. Indeed, whenever trial court opinions are cited, they are often discounted as aberrant or transitory, in part because they are known to be unrepresentative!
Why, you might wonder, is the convention of data-driven-scholarship a particular problem in quantitative empirical work? ELS’s detractors make three interrelated claims:
- If the availability of the data drives your research question, it’s not obvious that the questions you are asking are interesting, nor that they related to a real-world problem of note;
- ELS, like law and economics, appears to feel like science — it tends to want to displace “softer” disciplines, such as the unpacking and categorization of appellate court opinions. Because legal scholars lack empirical training, the black box effect looms large. Therefore, it’s important, but hard, to be especially on guard against shoddy work; and
- It’s normal & appropriate to read appellate opinions to know what the “law” is – because the “law” isn’t really (or mostly) about how litigants dispose of their cases, but rather how judges justify the imposition of force.
The first claim is true, though of course it would sweep almost all legal scholarship under its rug. (Which wouldn’t be so bad! Most legal scholarship, like most judicial opinions and all blog posts, influences no one and matters not a whit to the turning of the world.) The empirical work I do usually involves hand-coding of datasets, in part to avoid precisely this problem.
The second claim is harder to operationalize – but has bite. True: the rhetoric of empirical legal studies tends to be triumphalist: “I have found a statistically significant result: therefore, you are wrong!” (Or, better by far, “I haven’t, so you may be right!”) But isn’t arrogance about method, subject and approach a sin of almost all legal scholarship? As for the argument about training, the cost of learning how to become an educated reader of a regression isn’t terribly high, especially for a legal academic who has summers off.
The third claim to me feels like twaddle, with some minor exceptions. To me the more interesting questions about law turn on how citizens experience and understand legal practices and rules, not how they are crafted and justified in the tribunals above. Even if this is just a preference, I tend to think that the claim about what the law is represents exactly the data-driven ideology that Llewellyn called out. And at the very least, those who subscribe to the ideology of appellate opinion practice are just as likely to reach beyond their data to tell us what is happening in the world as are the new quantitative ELSers.