Category: Empirical Analysis of Law



As I previously have discussed here and here, I’ve been working on a project examining when trial courts write opinions. With the help of statistician co-authors, I have investigated trial court dockets, trying to account for various factors that might lead a contested matter to either be explained through a traditional written opinion or issued in a brief order. Our resulting draft, “Docketology, District Courts, and Doctrine”, is now available from SSRN or from Selected Works. Here is an abstract:

Empirical legal scholars have traditionally modeled judicial opinion writing by assuming that judges act rationally, seeking to maximize their influence by writing opinions in politically important cases. Support for this hypothesis has reviewed published opinions, finding that civil rights and other “hot” topics are more to be discussed than other issues. This orthodoxy comforts consumers of legal opinions, because it suggests that opinions are largely representative of judicial work.

The orthodoxy is substantively and methodologically flawed. This paper starts by assuming that judges are generally risk averse with respect to reversal, and that they provide opinions when they believe that their work will be reviewed by a higher court. Judges can control risk, and maximize leisure, by writing in cases that they believe will be appealed. We test these intuitions with a new methodology, which we call docketology. We have collected data from 1000 cases in 4 different jurisdictions. We recorded information about every judicial action over each case’s life.

Using a hierarchical linear model, our statistical analysis rejects the conventional orthodoxy: judges do not write opinions to curry favor with the public or with powerful audiences, nor do they write more when they are younger, seeking to advance their careers. Instead, judges write more opinions at procedural moments (like summary judgment) when appeal is likely and less opinions at procedural moments (like discovery) when it is not. Judges also write more in cases that are later appealed. This suggests that the dataset of opinions from the trial courts is significantly warped by procedure and risk aversion: we can not look at opinions to capture what the “Law” is.

These results have unsettling implications for the growing empirical literature that uses opinions to describe judicial behavior. It also challenges the meaning of doctrine, as we show that the vast majority of judicial work – almost 90% of substantive orders, and 97% of all judicial actions – are not fully reasoned, and are read only by the parties. Those rare orders that are explained by opinions are, at best, unrepresentative. At worst, they are true black sheep – representing moments and issues where the court is most obviously rejecting traditional patterns and analyses.

I am very interested in receiving comments on this paper, particularly before the late summer, when we plan to submit it to the law reviews!

[Nit-seekers beware: there is one typo in the SSRN abstract. (Don’t go find it, just trust me, it is there.) For what it is worth, I basically agree with Kevin Heller that SSRN should give users more control over author-submitted papers to make revision easier. ]


Neuroscience and Law

Jeffery Rosen has a fascinating article in this week’s New York Times Magazine. While the article is balanced and careful, the “buy me, read me” headlines and several of the researchers that Rosen quotes suggest that a law-and-neuroscience revolution is brewing. I want to add my voice to the skeptics that Rosen quotes, though with a different perspective. To my mind, recent findings in the field of neuroscience will change law only at the margins; and its main contribution will be to confirm the central tenets of legal realism, and will thus have only minor effects on most legal concerns.

Read More


Lewis Libby (66%) Guilty

Price for Lewis (Scooter) Libby Charges at

While Scott may not be “100% sold on either side” of the Libby trial, two-thirds of traders on the prediction markets think that he will be found guilty of lying on at least one charge against him. This is a notable upswing from last June, when I wrote about the market and its intricacies. I then pointed out that prices for these “conviction contracts” include a discount for the likelihood of a plea, so that in the months before trial, prices for conviction are likely to be depressed. I think that the rise in the price of Libby’s conviction stock demonstrates the point. Although there were few surprises at trial, traders raised the likelihood of conviction by over 20% after it began, representing the end of the plea discount and the market’s real estimation of the likelihood of conviction. Notably, traders don’t seem to think that a mistrial is terribly likely, although the likelihood of conviction has decreased from 75% at the beginning of the jury’s deliberations.

For what it is worth, I tend to agree with Scott that the length of the jury’s deliberations is a mark of seriousness and worth. If we wanted a quick and summary answer, we wouldn’t use a jury, we’d flip a coin.


Pie Charts: The Prime Evil

pie2.pngI’ve been busy working on edits for my recent paper, which attempts to present lots of data in relatively clear ways. I’ve gotten some great comments from readers. None more so than this one, in response to a proposed figure providing some descriptive statistics:

Pie charts are bad! They are ugly and provide the reader no visual assistance in comparing categories.

I had no idea if this was a generally accepted view among experts in the visual display of quantitative information. Extensive research suggested that it was:

One of the prevailing orthodoxies of this forum – one to which I whole-heartedly subscribe – is that pie charts are bad and that the only thing worse than one pie chart is lots of them.

The thing I don’t get is why: pie charts seem to be a very common form of data presentation; and folks are accustomed to measuring the area of slices of pie, so the visuals convey important data. What’s wrong with a slice of pie? (And, more importantly, are dot plots really better?)


Law Profs Who Code


Law Professors who write about the Internet tend to develop facts through a combination of anecdote and secondary-source research, through which information about the conduct of computer users, the network’s structure and architecture, and the effects of regulation on innovation are intuited, developed through stories, or recounted from others’ research. Although I think a lot of legal writing about the Internet is very, very good, I’ve long yearned for more “primary source” analysis.

In other words, there is room and need for Internet law scholars who write code. Although legal scholars aren’t about to break fundamental new ground in computer science, the hidden truths of the Internet don’t run very deep, and some very simple code can elicit some important results. Also, there is a growing cadre of law professors with the skills needed to do this kind of research. I am talking about a new form of empirical legal scholarship, and empiricists should embrace the perl script and network connection as parts of their toolbox, just as they adopted the linear regression a few decades ago.

I plan to talk about this more in a subsequent post or two, but for now, let me give some examples of what I’m describing. Several legal scholars (or people closely associated with legal scholarship) are pointing the way for this new category of “empirical Internet legal studies”.

  • Jonathan Zittrain and Ben Edelman, curious about the nature and extent of filtering in China and Saudi Arabia, wrote a series of scripts to “tickle” web proxies in those countries to analyze the amount of filtering that occurs.
  • Edelman has continued to engage in a particularly applied form of Internet research, for example see his work on spyware and adware.
  • Ed Felten—granted, a computer scientist not a law professor—and his graduate students at Princeton have investigated DRM and voting machines with a policy bent and a particular focus on applied, clear results. Although the level of technical sophistication found in these studies is unlikely to be duplicated in the legal academy soon, his methods and approaches are a model for what I’m describing.
  • Journalist Kevin Poulsen created scripts that searched MySpace’s user accounts for names and zip codes that matched the DOJ’s National Sex Offender Registry database, and found more than 700 likely matches.
  • Finally, security researchers have set up vulnerable computers as “honeypots” or “honeynets” on the Internet, to give them a vantage point from which to study hacker behavior.

What are other notable examples of EILS? Let’s keep with the grand Solovian tradition, and call this a Census. Is this sub-sub-discipline ready to take off, or should we mere lawyers leave the coding to the computer scientists?


Replicability, Exam Grading, and Fairness

Exam-Grade-2a.jpgWhat does it mean to grade fairly?

At my law school, and presumably elsewhere, law students aggrieved by a grade can petition that it be changed. Such petitions are often granted in the case of mathematical error, but usually denied if the basis is that on re-reading, the professor would have reached a different result. The standard of review for such petitions is something like “fundamental fairness.” In essence, replicability is not an integral component of fundamental fairness for these purposes.

Law students may object to this standard, and its predictable outcome, asserting that if the grader can not replicate his or her outcomes when following the same procedure, then the total curve distribution is arbitrary. On this theory, a student at the least should have the right to a new reading of their test, standing alone and without the time-pressure that full-scale grading puts on professors.

To which the response is: grading is subjective, and not subject to scientific proof. Moreover, grades don’t exist as platonic ideals but rather distributions between students: only when reading many exams side by side can such a ranking be observed. We wouldn’t even expect that one set of rankings would be very much like another: each is sort of like a random draw of a professor’s gut-reactions to the test on that day.

This common series of arguments tends to engender cynicism among high-GPA and low-GPA students alike. To the extent that law school grading is underdetermined by work, smarts and skill, it is a bit of a joke. The importance placed on these noisy signals by employers demonstrates something fundamentally bitter about law – the power of deference over reason.

Read More


Can NVivo Qualitative Empirical Software Help Manage Oceans Of Research?

One of the real challenges for a legal scholar (and probably researchers in many other social science disciplines as well) is figuring out what to do with all those interesting articles you read. Do you make notebooks organized by topic? If so, what happens when a piece has something important to say on multiple topics? Do you create index cards, or their digital equivalents, with relevant quotes? Or, like me, do you find yourself rediscovering the wheel several times – putting an article aside in stack on day one, and rediscovering it on Lexis or Westlaw four months later when you’re searching for a different issue?

I find that keeping control of existing literature, a critical process for those who publish in law reviews (which demand a footnote to support even the most mundane statements), turns out to be a burdensome and sometimes unsuccessful pursuit. As a result, I’m very intrigued by the idea of using NVivo to help.

What is NVivo? It’s a leading qualitative empirical research software. Yes, Virginia, I did say qualitative. As many folks know, one of my biggest beefs with Empirical Legal Studies is that some of its followers have marginalized qualitiative research – so much that many people with only a passing awareness of ELS believe that all empirical work is quantitative. That discussion is for another day, however. The point is that qual researchers use software to help them keep track of their data…which is to say, their texts. My understanding of NVivo – formerly known as NUD*IST – is that you can take texts (like law review articles) and drop them into the software. You can then create coding fields, and mark selected text as part of such fields. (A discussion of the capacities of qual software is here.) For example, if one were studying the way that courts discuss victims in rape cases, and had created a sample for investigation, one might load the selected cases into NVivo. As the researcher creates particular fields – for example “victim dressed provocatively”, “victim drinking”, “victim previously worked as prostitute” (as well as “circuit court”, “appellate court”, “female judge”) – she can then mark text in each case that would fit into the field. This allows her, at a later point, to do targeted searches for particular marked themes – and also allows her to subdivide by the traits of the cases. Thus, she can identify all the decisions by female judges that identify females as victims, and break them out by year.

I wonder whether many legal scholars who don’t do qualitative work could benefit from this software simply by using it as a way of containing, coding and organizing all the articles they read in the course of their literature review. I haven’t heard of anyone doing this, but it seems like it might make a lot of sense – particularly for somewhat disorganized researchers. It might not take advantage of all the power of NVivo, but it could be the equivalent of the smartest filing system ever created.

Does anyone have experience with NVivo, or other similar software (like Atlas), that might shed light on this? By the way, many schools have site licenses for this software, so many of those interested in trying this out can do so without spending a dime.


A minimum wage field experiment

Thanks to Dan for inviting me to guest blog!

Tyler Cowen suggests that the expected increase in the minimum wage may serve as a useful “controlled experiment,” at least if the increase applies to Northern Mariana but not to American Samoa. A commenter points out that it’s not a well controlled experiment, because the two territories are not identical. This point rehearses a familiar challenge for empirical legal analysis: Legal scholars don’t have the luxury of randomized studies. Even natural experiments rarely provide conclusive evidence of policy effects.

But we could have randomized studies (or so I will argue in a paper that I am working on). John List is a leader among those who do “field tests” rather than using laboratory experiments or relying on other econometric techniques. (I refer of course to John List the economist, not John List the family murderer, although the boundary between these occupations has allegedly blurred recently.) We could do field tests in law, if only legislatures would cooperate.

An explanation, after the jump.

Read More

Song of Jersey City

PATH Map.jpg

Rick Garnett recently wrote on “cities’ hipness competition.” According to a recent article in New York Magazine, my urban home (Jersey City) has recently won some prize:

To live [in New York now] is to endure a gnawing suspicion that somebody, somewhere, is marveling and reveling a little more successfully than you are. That they’re paying less money for a bigger apartment with more-authentic details on a nicer block closer to cuter restaurants and still-uncrowded bars and hipper galleries that host better parties with cooler bands than yours does, in an area that’s simultaneously a portal to the future (tomorrow’s hot neighborhood today!) and a throwback to an untainted past (today’s hot neighborhood yesterday!). And you know what? Someone is. And you know what else? Right now, that person just might be living in Jersey City.

It’s not just Tyler Cowen who’s rescuing New Jersey from punchline status–even the uberhip NYM is recognizing us (even if we’re shunned by NYC Bloggers). Our hospitals may be closing, but at least we’ve got a hot arts scene.

Of course, the NYM piece focuses not on all of the JC, but only on the “downtown” close to the Hudson waterfront. I live a bit further down the PATH line, in Journal Square. I think a comparison between the two areas may help us answer Rick’s question: “what law can do — e.g., zoning laws, liquor licensing, etc. — to make cities / metro areas more (or less) attractive to the young (or the old, for that matter)”? Can big urbanism work?

Read More

From the New Property to the New Responsibility

apple small.jpgJust as Charles Reich was a premier theorist of rights to government largesse, Peter Schuck and Richard Zeckhauser are leading exponents of the responsibilities it entails. In Targeting Social Programs, S&Z focus on the denial of benefits to “bad bets” and “bad apples:”

Bad bets are individuals who are likely to benefit little from social resources relative to other [beneficiaries]. . . . Bad apples are individuals whose irresponsible, immoral, or illegal behavior in the past—and predictably, in the future as well—marks them as unsuitable to receive the benefits of social programs.

This may sound a bit cold-hearted at first, but S&Z make a good case that, behind a veil of ignorance, we’d quite sensibly allocate resources to, say, the transplant recipient who is most likely to benefit, rather than the one who has been on the wait list the longest. They also show how often “bad apples'” worst effects are on the disadvantaged citizens near them. (For an example, see Kahan and Meares on anti-loitering ordinances.)

The West Virginia Medicaid program provides an interesting case study of “bad apple screening.” Consider the fate of one beneficiary who refuses to sign a “health responsibility contract:”

Mr. Johnson. . . goes to a clinic once a month for diabetes checkups. Taxpayers foot the bill through Medicaid . . . [b]ut when doctors urged him to mind his diet, “I told them I eat what I want to eat and the hell with them. . . . I’ve been smoking for 50 years — why should I stop now? . . . This is supposed to be a free world.”

Traditionally, there was little Medicaid could do to encourage compliance. But now, “[u]nder a reorganized schedule of aid, the state, hoping for savings over time, plans to reward “responsible” patients with significant extra benefits or — as critics describe it — punish those who do not join weight-loss or antismoking programs, or who miss too many appointments, by denying important services.” But as the article notes, “Somewhat incongruously, [Johnson] appears to be off the hook: as a disabled person he will be exempt under the rules.”

Critics claim the program is unduly intrusive: “What if everyone at a major corporation were told they would lose benefits if they didn’t lose weight or drink less?” asked one doctor. Certainly in some manifestations it could be; consider this 1997 proposal by Judge John Marshall Meisburg:

Congress should . . . consider legislation stipulating that no one can be granted disability by SSA if s/he continues to smoke against the advice of his physician, and smoking is a factor material to the disability, because such claimants are bringing illness and disability upon themselves. Such a law would reduce the burden of proof now needed to deny benefits to persons who fail to heed their doctors’ advice, and would dovetail with legislation just passed by Congress to abolish disability benefits for persons addicted to drug and alcohol. In many cases, smoking is akin to “contributory negligence” and the SSA law should recognize it as such. [From Federal Lawyer, 44-APR FEDRLAW 56 on Westlaw.]

I think S&Z frame the debate in a nuanced enough way to avoid this kind of draconian proposal. But I do have a few quibbles with the framing of their work, if not its substance.

Read More