Category: Empirical Analysis of Law


CELS VIII: Data is Revealing, Part 1.


"If you are going to mine my data, at least have the courtesy of displaying predictive probabilities!"

“If you are going to mine my data, at least have the courtesy of displaying predictive probabilities!”

[This is part 1 of my recap of the Penn edition of CELS, promised here.  For previous installments in the CELS recap series, see CELS III, IV, V, and VI, VII.]

Barry Schwartz might’ve designed the choice set facing me at the opening of CELS. Should I go to Civil Procedure I (highlighted by a Dan Klerman paper discussing the limits of Priest-Klein selection), Contracts I (where Yuval Feldman et al. would present on the relationship between contract clause specificity and compliance), on Judicial Decisionmaking and Settlement (another amazing Kuo-Chang Huang paper). [I am aware, incidentally, that for some people this choice would be Morton’s. But those people probably weren’t the audience for this post, were they.] I bit the bullet and went to Civ Pro, on the theory that it’d be a highly contentious slugfest between heavyweights in the field, throwing around words like “naive” and “embarrassing.”  Or, actually, I went hoping to learn something from Klerman, which I did. The slugfest happened after he finished.

In response to a new FJC paper on pleading practices, a discussant and a subsequent presenter criticized the FJC’s work on Twiqbal. The discussant argued that the FJC’s focus on the realities of lawyers’ practice was irrelevant to the Court’s power-grab in Twombly, and that pleading standards mattered infinitely more than pleading practice.  The presenter argued that the FJC committed methodological error in their important 2011 survey, and that their result (little effect) was misleading. The ensuing commentary was not restrained. Indeed, it felt a great deal like the infamous CELS death penalty debate from 2008. One constructive thing did come out of the fire-fight: the FJC’s estimable Joe Cecil announced that he would be making the FJC’s Twombly dataset available to all researchers through Vandy’s Branstetter program. We’ll all then be able to replicate the work done, and compare it to competing coding enterprises. Way to go, Joe!

But still, it was a tense session.  As it was wrapping up, an economically-trained empiricist in the room commented how fun he had found it & how he hoped to see more papers on the topic of Twombly in the future. I’d been silent to that point, but it was time to say something.  Last year in this space I tried being nice: “My own view would go further: is Twiqbal’s effect as important a problem as the distribution of CELS papers would imply?” This year I was, perhaps impolitically, more direct.

I conceded that analyzing the effect of Twombly/Iqbal wasn’t a trivial problem. But if you had to make a list of the top five most important issues in civil procedure that data can shed light on, it wouldn’t rank.* I’m not sure it would crack the top ten.  Why then have Twiqbal papers eaten market share at CELS and elsewhere since 2011? Some hypotheses (testable!) include: (1) civil procedure’s federal court bias; (2) giant-killing causes publication, and the colossi generally write normative articles praising transsubstantive procedure and consequently hate Twombly; (3) network effects; and (4) it’s where the data are. But these are bad reasons. Everyone knows that there is too much work on Twombly. We should stop spending so much energy on this question. It is quickly becoming a dead end.

So I said much of that and got several responses. One person seemed to suggest that a good defense of Twiqbal fixation was that it provided a focal point to organize our research and thus build an empirical community. Another suggested that even if law professors were Twiqbal focused, the larger empirical community was not (yet) aware of the importance of pleadings, so more attention was beneficent. And the rest of folks seemed to give me the kind of dirty look you give the person who blocks your view at a concert. Sit down! Don’t you see the show is just getting started?

Anyway, after that bit of theatre, I was off to a panel on Disclosure. I commented (PPT deck) on Sah/Lowenstein, Nothing to Declare: Mandatory and Voluntary Disclosure leads advisors to avoid conflicts of interestThis was a very, very good paper, in the line of disclosure papers I’ve previously blogged here. The innovation was that advisors were permitted to walk away from conflicts instead of being assigned to them immutably. This one small change cured disclosure’s perverse effect. Rather than being morally licensed by disclosure to lie, cheat and steal, advisors free to avoid conflicts were chastened by disclosure just as plain-vanilla Brandeisian theory would’ve predicted.   In my comments, I encouraged Prof. Sah to think about what happened if advisors’ rewards in the COI were returned to a third party instead of to them personally, since I think that’s the more legally-relevant policy problem. Anyway, definitely worth your time to read the paper.

Then it was off to the reception. Now, as our regular readers know, the cocktail party/poster session is a source of no small amount of stress. On the one hand, it’s a concern for the organizers. Will the food be as good as the legendary CELS@Yale? The answer, surprisingly, was “close to it”, headlined by some grapes at a cheese board which were the size of small apples and tasted great.  Also, very little messy finger food, which is good because the room is full of the maladroit.  But generally, poster sessions are terribly scary for those socially awkward introverts in the crowd. Which is to say, the crowd. In any event, I couldn’t socialize because I had to circle the crowd for you. Thanks for the excuse!

How about those posters?  I’ll highlight two. The first was a product of Ryan Copus and Cait Unkovic of Bolt’s JSP program. They automated text processing of appellate opinions and find significant judge-level effects on whether the panel reverses the district court’s opinion, as well as strong effects for the decision to designate an opinion for publication in the first instance. That was neat. But what was neater was the set of judicial base cards, complete with bubble-gum and a judge-specific stat pack, that they handed out.  My pack included Andrew Kleinfeld, a 9th circuit judge who inspired me to go to law school.  The second was a poster on the state appellate courts by Thomas Cohen of the AO. The noteworthy findings were: (1) a very low appeal-to-merits rate; and (2) a higher reversal rates for plaintiff than defendant wins at trial. Overall, the only complaint I’d make about the posters was that they weren’t clearly organized in the room by topic area, which would have made it easier to know where to spend time.  Also, the average age of poster presenters was younger than the average age of presenters of papers, while the average quality appeared as high or higher. What hypotheses might we formulate to explain that distribution?

That was all for Day 1. I’ll write about Day 2, which included a contracts, international law, and legal education sessions,  in a second post.


*At some point, I’ll provide a top ten list.  I’m taking nominations.  If it has federal court in the title, you are going to have to convince me.


Stanford Law Review Online: Privacy and Big Data

Stanford Law Review

The Stanford Law Review Online has just published a Symposium of articles entitled Privacy and Big Data.

Although the solutions to many modern economic and societal challenges may be found in better understanding data, the dramatic increase in the amount and variety of data collection poses serious concerns about infringements on privacy. In our 2013 Symposium Issue, experts weigh in on these important questions at the intersection of big data and privacy.

Read the full articles, Privacy and Big Data at the Stanford Law Review Online.



The Dignity of the Minimum Wage?

[A brief note of apology: it’s been a terrible blogging summer for me, though great on other fronts.  I promise I’ll do better in the coming academic year. In particular, I’d like to get back to my dark fantasy/law blogging series. If you’ve nominations for interviewees, email me.]

WorkDetroitThis is one I’ve been meaning to write for a while.

One of the major lessons of the cultural cognition project is that empirical arguments are a terrible way to resolve value conflicts. On issues as diverse as the relationship between gun ownership and homicide rates, the child-welfare effects of gay parenting, global warming, and consent in rape cases, participants in empirically-infused politics behave as if they are spectators at sporting events. New information is polarized through identity-protective lenses; we highlight those facts that are congenial to our way of life and discounts those that are not; we are subject to naive realism.  It’s sort of dispiriting, really.  Data can inflame our culture wars.

One example of this phenomenon is the empirical debate over minimum wage laws. As is well known, there is an evergreen debate in economics journals about the policy consequences which flow from a wage floor.  Many (most) economists argue that the minimum wage retards growth and ironically hurts the very low-wage workers it is supposed to hurt. Others argue that the minimum wage has the opposite effect. What’s interesting about this debate -to me, anyway- is that it seems to bear such an orthogonal relationship to how the politics of the minimum wage play out, and the kinds of arguments that persuade partisans on one side or another. Or to put it differently, academic liberals in favor of the minimum wage have relied on regression analyses, but I don’t think they’ve persuaded many folks who weren’t otherwise disposed to agree with them. Academic critics of the minimum wage too have failed to move the needle on public opinion, which (generally) is supportive of a much higher level of minimum wage than is currently the law.

How to explain this puzzle?  My colleague Brishen Rogers has a terrific draft article out on ssrn, Justice at Work: Minimum Wage Laws and Social Equality. The paper urges a new kind of defense of minimum wages, which elides the empirical debate about minimum wages’ effect on labor markets altogether. From the abstract:

“Accepting for the sake of argument that minimum wage laws cause inefficiency and unemployment, this article nevertheless defends them. It draws upon philosophical arguments that a just state will not simply redistribute resources, but will also enable citizens to relate to one another as equals. Minimum wage laws advance this ideal of “social equality” in two ways: they symbolize the society’s commitment to low-wage workers, and they help reduce work-based class and status distinctions. Comparable tax-and-transfer programs are less effective on both fronts. Indeed, the fact that minimum wage laws increase unemployment can be a good thing, as the jobs lost will not always be worth saving. The article thus stands to enrich current increasingly urgent debates over whether to increase the minimum wage. It also recasts some longstanding questions of minimum wage doctrine, including exclusions from coverage and ambiguities regarding which parties are liable for violations.”

I’m a huge fan of Brishen’s work, having been provoked and a bit convinced by his earlier work (here) on a productive way forward for the union movement. What seems valuable in this latest paper is that the minimum wage laws are explicitly defended with reference to a widely shared set of values (dignity, equality). Foregrounding such values I think would increase support for the minimum wage among members of the populace.  The lack of such dignitary discussions in the academic debate to date has level the minimum wage’s liberal defenders without a satisfying and coherent ground on which to stand. Worth thinking about in the waning hours of Labor’s day.




Brian Tamanaha’s Straw Men (Part 2): Who’s Cherry Picking?

(Reposted from Brian Leiter’s Law School Reports)

BT Claim 2:  Using more years of data would reduce the earnings premium

BT Quote: There is no doubt that including 1992 to 1995 in their study would measurabley reduce the ‘earnings premium.'” 

Response:  Using more years of historical data is as likely to increase the earnings premium as to reduce it

We have doubts about the effect of more data, even if Professor Tamanaha does not.

Without seeing data that would enable us to calculate earnings premiums, we can’t know for sure if introducing more years of comparable data would increase our estimates of the earnings premium or reduce it.

The issue is not simply the state of the legal market or entry level legal hiring—we must also consider how our control group of bachelor’s degree holders (who appear to be similar to the law degree holders but for the law degree) were doing.   To measure the value of a law degree, we must measure earnings premiums, not absolute earnings levels.

As a commenter on Tamanaha’s blog helpfully points out:

“I think you make far too much of the exclusion of the period from 1992-1995. Entry-level employment was similar to 1995-98 (as indicated by table 2 on page 9).

But this does not necessarily mean that the earnings premium was the same or lower. One cannot form conclusions about all JD holders based solely on entry-level employment numbers. As S&M’s data suggests, the earnings premium tends to be larger during recessions and their immediate aftermath and the U.S. economy only began an economic recovery in late 1992.

Lastly, even if you are right about the earnings premium from 1992-1995, what about 1987-91 when the legal economy appeared to be quite strong (as illustrated by the same chart referenced above)? Your suggestion to look at a twenty year period excludes this time frame even though it might offset the diminution in the earnings premium that would allegedly occur if S&M considered 1992-95.”

There is nothing magical about 1992.  If good quality data were available, why not go back to the 1980s or beyond?   Stephen Diamond and others make this point.

The 1980s are generally believed to be a boom time in the legal market.  Assuming for the sake of the argument that law degree earnings premiums are pro-cyclical (we are not sure if they are), inclusion of more historical data going back past 1992 is just as likely to increase our earnings premium as to reduce it.  Older data might suggest an upward trend in education earnings premiums, which could mean that our assumption of flat earnigns premiums may be too conservative. Leaving aside the data quality and continuity issues we discussed before (which led us to pick 1996 as our start year), there is no objective reason to stop in the early 1990s instead of going back further to the 1980s.

Our sample from 1996 to 2011 includes both good times and bad for law graduates and for the overall economy, and in every part of the cycle, law graduates appear to earn substantially more than similar individuals with only bachelor’s degrees.




This might be as good a place as any to affirm that we certainly did not pick 1996 for any nefarious purpose.  Having worked with the SIPP before and being aware of the change in design, we chose 1996 purely because of the benefits we described here.  Once again, should Professor Tamanaha or any other group wish to use the publicly available SIPP data to extend the series farther back, we’ll be interested to see the results.


Brian Tamanaha’s Straw Men (Part 1): Why we used SIPP data from 1996 to 2011

(Reposted from Brian Leiter’s Law School Reports)


BT Claim:  We could have used more historical data without introducing continuity and other methodological problems

BT quote:  “Although SIPP was redesigned in 1996, there are surveys for 1993 and 1992, which allow continuity . . .”

Response:  Using more historical data from SIPP would likely have introduced continuity and other methodological problems

SIPP does indeed go back farther than 1996.  We chose that date because it was the beginning of an updated and revitalized SIPP that continues to this day.  SIPP was substantially redesigned in 1996 to increase sample size and improve data quality.  Combining different versions of SIPP could have introduced methodological problems.  That doesn’t mean one could not do it in the future, but it might raise as many questions as it would answer.

Had we used earlier data, it could be difficult to know to what extent changes to our earnings premiums estimates were caused by changes in the real world, and to what extent they were artifacts caused by changes to the SIPP methodology.

Because SIPP has developed and improved over time, the more recent data is more reliable than older historical data.  All else being equal, a larger sample size and more years of data are preferable.  However, data quality issues suggest focusing on more recent data.

If older data were included, it probably would have been appropriate to weight more recent and higher quality data more heavily than older and lower quality data.  We would likely also have had to make adjustments for differences that might have been caused by changes in survey methodology.  Such adjustments would inevitably have been controversial.

Because the sample size increased dramatically after 1996, including a few years of pre 1996 data would not provide as much new data or have the potential to change our estimates by nearly as much as Professor Tamanaha believes.  There are also gaps in SIPP data from the 1980s because of insufficient funding.

These issues and the 1996 changes are explained at length in the Survey of Income and Program Participation User’s Guide.

Changes to the new 1996 version of SIPP include:

Roughly doubling the sample size

This improves the precision of estimates and shrinks standard errors

Lengthening the panels from 3 years to 4 years

This reduces the severity of the regression to the median problem

Introducing computer assisted interviewing to improve data collection and reduce errors or the need to impute for missing data

Introducing oversampling of low income neighborhoods
This mitigates response bias issues we previously discussed, which are most likely to affect the bottom of the distribution
New income topcoding procedures were instituted with the 1996 Panel
This will affect both means and various points in the distribution
Topcoding is done on a monthly or quarterly basis, and can therefore undercount end of year bonuses, even for those who are not extremely high income year-round

Most government surveys topcode income data—that is, there is a maximum income that they will report.  This is done to protect the privacy of high-income individuals who could more easily be identified from ostensibly confidential survey data if their incomes were revealed.

Because law graduates tend to have higher incomes than bachelor’s, topcoding introduces downward bias to earnings premiums estimates. Midstream changes to topcoding procedures can change this bias and create problems with respect to consistency and continuity.

Without going into more detail, the topcoding procedure that began in 1996 appears to be an improvement over the earlier topcoding procedure.

These are only a subset of the problems extending the SIPP data back past 1996 would have introduced.  For us, the costs of backfilling data appear to outweigh the benefits.  If other parties wish to pursue that course, we’ll be interested in what they find, just as we hope others were interested in our findings.


Brian Tamanaha’s Straw Men (Overview)

(Cross posted from Brian Leiter’s Law School Reports)

Brian Tamanaha previously told Inside Higher Education that our research only looked at average earnings premiums and did not consider the low end of the distribution.  Dylan Matthews at the Washington Post reported that Professor Tamanaha’s description of our research was “false”. 

In his latest post, Professor Tamanaha combines interesting critiques with some not very interesting errors and claims that are not supported by data.   Responding to his blog post is a little tricky as his ongoing edits rendered it something of a moving target.  While we’re happy with improvements, a PDF of the version to which we are responding is available here just so we all know what page we’re on.

Stephen Diamond explains why Tamanaha apparently changed his post: Ted Seto and Eric Rasmusen expressed concerns about Tamanaha’s use of ad hominem attacks.

Some of Tamanaha’s new errors are surprising, because they come after an email exchange with him in which we addressed them.  For example, Tamanaha’s description of our approach to ability sorting constitutes a gross misreading of our research.  Tamanaha also references the wrong chart for earnings premium trends and misinterprets confidence intervals.  And his description of our present value calculations is way off the mark.

Here are some quick bullet point responses, with details below in subsequent posts:

  • Forecasting and Backfilling
    • Using more historical data from SIPP would likely have introduced continuity and other methodological problems
    • Using more years of data is as likely to increase the historical earnings premium as to reduce it
    • If pre-1996 historical data finds lower earnings premiums, that may suggest a long term upward trend and could mean that our estimates of flat future earnings premiums are too conservative and the premium estimates should be higher
    • The earnings premium in the future is just as likely to be higher as it is to be lower than it was in 1996-2011
    • In the future, the earnings premium would have to be lower by **85 percent** for an investment in law school to destroy economic value at the median
  • Data sufficiency
    • 16 years of data is more than is used in similar studies to establish a baseline.  This includes studies Tamanaha cited and praised in his book.
    • Our data includes both peaks and troughs in the cycle.  Across the cycle, law graduates earn substantially more than bachelor’s.
  • Tamanaha’s errors and misreading
    • We control for ability sorting and selection using extensive controls for socio-economic, academic, and demographic characteristics
    • This substantially reduces our earnings premium estimates
    • Any lingering ability sorting and selection is likely offset by response bias in SIPP, topcoding, and other problems that cut in the opposite direction
    • Tamanaha references the wrong chart for earnings premium trends and misinterprets confidence intervals
    • Tamanaha is confused about present value, opportunity cost, and discounting
    • Our in-school earnings are based on data, but, in any event, “correcting” to zero would not meaningfully change our conclusions
  • Tamanaha’s best line
    • “Let me also confirm that [Simkovic & McIntyre’s] study is far more sophisticated than my admittedly crude efforts.”

Have Presidents Gotten Better at Picking Ideologically-Compatible Justices?

Do Justices vote independently of all political forces surrounding their appointments? My earlier post discusses how, even in recent decades, Justices’ votes have been surprisingly independent of the ideologies of Senates to which they were nominated. Even so, it may be that presidents fared better than the Senate and recently enhanced their ability to appoint ideologically-compatible Justices.

History is rife with examples of Justices who disappointed their appointing presidents.   As recounted by Henry Abraham, Teddy Roosevelt complained vociferously about Justice Holmes’ ruling in Northern Securities, Truman called Justice Clark his “biggest mistake,” and Eisenhower also referred to Justices Warren and Brennan as “mistakes.”  My earlier study finds frequent grounds for presidential disappointment, based on voting records for eighty-nine Justices over a 172-year period. Just under half of these Justices voted with appointees of the other party most of the time. Still, of the last twelve Justices, only two, Stevens and Souter, aligned most often with appointees of the other party. This low number calls into question whether the frequency of presidential disappointments has diminished recently.

My recent paper identifies change over time using regression analysis and more nuanced measures of presidential ideology. The analysis shows ideologies of appointing presidents did not significantly predict Justices’ votes before the 1970s, but they gained significant predictive power thereafter. This enhanced success coincides with Presidents Nixon’s and Reagan’s efforts to prioritize ideology in appointments to the bench. While earlier presidents did not uniformly ignore nominees’ ideology, they lacked modern technological resources. By the Reagan administration, computerized databases allowed presidential aides to quickly assemble and analyze virtually all of a nominee’s past writings. The improved information may have enabled presidents to better anticipate nominees’ future rulings.


The Senate’s Influence over Supreme Court Appointments

Thanks, Sarah, for the warm welcome. It is a pleasure to guest blog this month.

With pundits already speculating about President Obama’s next Supreme Court nominee, it seems a good time to discuss relationships between political forces surrounding Supreme Court appointments and Justices’ decisions. Justices sometimes disappoint their appointing presidents, and ideologically-distant Senates are often blamed for presidents’ “mistakes.” For example, David Souter and John Paul Stevens turned out to be far more liberal than the Republican presidents who appointed them (Bush I and Ford, respectively). These presidents both faced very liberal Senates when they selected Souter and Stevens.

Are nominees like Souter and Stevens anomalies or part of a larger pattern of senatorial constraint? My recent article in the Hastings Law Journal offers the first empirical analysis of the Senate’s role in constraining presidents’ choices of Supreme Court nominees over an extended period. It considers ideologies of Senates faced by nominating presidents and measures whether the ideologies of these Senates predict Justices’ voting behavior. The analysis substantially qualifies earlier understandings of senatorial constraint.

Earlier empirical studies consider only limited numbers of recent nominees (see article pp. 1235-39). They suggest that the Senate has constrained presidents’ choices, and many scholars theorize that the Senate has enhanced its role in the appointments process since the 1950s. Analysis of a larger group of nominees shows the Senate’s ideology has had significant predictive power over Justices’ votes in only two isolated historical periods. Senatorial ideology was last significant in the 1970s, shortly after the filibuster of Abe Fortas’s nomination to be Chief Justice, but then it actually lost significance after the Senate rejected Bork in 1987.