Should Empirical Legal Scholars Have Special Responsibilities?

Before delving into the substance of my first post, I wanted to thank the crew at Concurring Opinions for inviting me to guest blog this month.

Recently, I have been thinking about whether empirical legal scholars have or should have special ethical responsibilities. Why special responsibilities? Two basic reasons. First, nearly all law reviews lack formal peer review. The lack of peer review potentially permits dubious data to be reported without differentiation alongside quality data. Second, empirical legal scholarship has the potential to be extremely influential on policy debates because it provides “data” to substantiate or refute claims. Unfortunately, many consumers of empirical legal scholarship — including other legal scholars, practitioners, judges, the media, and policy makers — are not sophisticated in empirical methods. Even more importantly, subsequent citations of empirical findings by legal scholars rarely take care to explain the study’s qualifications and limitations. Instead, subsequent citations often amplify the “findings” of the empirical study by over-generalizing the results. 

My present concern is about weak data. By weak data, I don’t mean data that is flat out incorrect (such as from widespread coding errors) or that misuses empirical methods (such as when the model’s assumptions are not met). Others previously have discussed issues relating to incorrect data and analysis in empirical legal studies. Rather, I am referring to reporting data that encourages weak or flawed inferences, that is not statistically significant, or that is of extremely limited value and thus may be misused. The precise question I have been considering is under what circumstances one should report weak data, even with an appropriate explanation of the methodology used and its potential limitations. (A different yet related question for another discussion is whether one should report lots of data without informing the reader which data the researcher views as most relevant. This scattershot approach has many of the same concerns as weak data.)

To take a contrived example, let’s say a researcher is relying in part upon old data and has reasons to believe that if more recent data were gathered and analyzed, the results would be different. Is it okay to publish results from the old data with a disclaimer saying that new data may yield different findings? To take another contrived example, let’s say a study finds as a secondary finding that 6 out of 10 randomly selected cases found X. For empiricists interested in statistical significance, there is almost nothing valuable to be said about the population from such a small sample size. Would it be appropriate to report that 60% of the sampled cases found X? Would it be sufficient if the article included a paragraph disclosing that the sample size was too small for purposes of making inferences? Some may argue that law reviews can figure this out. But weak data may be presented in an article that includes other solid data. While student editors may be able to identify articles relying upon solely weak data, they likely have more difficulty separating those that report both strong and weak data.

In many cases, I don’t believe transparency about the study’s methodology and their limitations is sufficient. Transparency is a bedrock requirement of solid empirical work. Furthermore, good empirical scholarship notes significant real or potential limitations. Readers of law review articles may unknowingly misuse the reported data because they lack an appreciation of the limitations. There is no peer review filter to remove portions of articles with weak data.  Student editors are unlikely to insist upon removal during the editing process.

So where do I come out? Based upon the non-peer reviewed law review system we have now, my current thinking is that sometimes a researcher should report weak data, and sometimes she shouldn’t. One guidepost to be used is peer review. While not every legal scholar is familiar with the criteria for peer review, a researcher should attempt to evaluate whether the data would be published through a peer review process. When in doubt, consult more experienced colleagues. To me, when the data in question supports a personal, normative view of the scholar, the scholar should be cautious in reporting it because the scholar can be vulnerable to undercounting the data’s weaknesses. 

The nature and timing of the issue being studied matters too. The scholar should err on the side of caution if she knows of amicus briefs in a pending case that would likely cite to the data. Even putting aside the potential for normative biases, scholars often want to include all of the data they have spent time gathering. Because they want credit for their hard work, they may be less willing to delete weak data than someone like a peer review referee who is unaffiliated with the project.  (Of course, empirical legal scholars should also consider submitting to peer-reviewed journals, either as a substitute for student-edited law reviews or as a companion to a paper published in a student-edited law review.)

Another guidepost is to consider if the weak data is the best data that the researcher can reasonably obtain and analyze under the available resources and constraints. When it is the best available data, this factor weighs toward reporting. Furthermore, if the weaker data is consistent with other, more solid data, and the researcher intends to later supplement the findings, perhaps reporting is appropriate. Sometimes reporting weak or preliminary data can spur further, more solid research. This may outweigh concerns about potential misunderstandings of the data.  And sometimes weak data is just one data point among many. Then, the risk that the data will be misused is smaller.

To reduce the possibility of misuse or misunderstanding, a heightened disclosure requirement is prudent. If a decision is made to include the weak data, a very explicit warning for the reader should be included. Scholars need to think more carefully about how to disclose uncertainty about the relevance of weak data.

You may also like...

6 Responses

  1. anon says:

    I think that the best solution is not to publish empirical research in student-edited, non-peer reviewed journals. It is not reasonable to assume that law students lacking empirical training will be able to discern the quality of an empirical study. Inevitably, they will pick articles with sexy topics or conclusions, regardless of the strength of the data.

  2. TJ says:


    I’m going to be less generous than you and say that, in many instances where I see what you describe as misuse of weak data by others, the misuse is likely intended by the author. That is, the most salient example of this problem to me is when an author reports the weak data, makes a bunch of policy-salient inferences prominently in the abstract and introduction, and then somewhere in the middle lists all the limitations and alternative inferences that can be drawn. If this kind of intentional sleight-of-hand is the core example of the problem, I don’t think it is a “special” obligation of an empirical researcher to refrain from it.

  3. ctr says:

    Most of the analysis by Schwartz and Anon seems equally applicable to non-empirical research. I see hasty generalizations and weak arguments under-determined by the available evidence in all sorts of scholarship.

    The distinguishing factor that Schwartz cites — that empirical research may have a disproportionate impact — is interesting. Perhaps, ceteris parabis, that’s an argument for more empirical scholarship, assuming that its not junk on the median.

    It bears emphasis that the alternative to weak data is often rank speculation. The courts and policymakers (and scholars) ROUTINELY rely on raw assumptions that align with their policy preferences or status quo biases. Even weak data — an accumulation of anecdotes — can light a candle to the darkness. Considering your 6/10 example — that finding could have real epistemic value if the alternative hypothesis is that 0% or 10% of the cases do X. (Assuming random sampling, etc.) Validity and statistical significance are relative concepts, and we are in a world of scarcity.

    Finally, anyone who has published in peer reviewed journals can attest to the large variance in the quality of peer reviewers. Frankly, the process suffers from a huge sampling problem itself — one or two hand-picked reviewers purport to speak for the rigor and other values of the entire field of hundreds of scholars, and thus get to be the secret censors? (Want to figure a p-value on that?)

    I have seen peer reviewers of empirical papers that couldn’t help but spend page after page arguing about the value-laden and political implications of the work, and with those glasses firmly affixed trying to also evaluate the scientific rigor of the study. This is just to say that we are all human; I may be guilty of the same.

    Even if the two censors are representative of the field, that process causes many false positives and false negatives too, errors which are not random but are biased towards paradigm-maintenance. Perhaps a politically-charged field like ours is not one where censorship by peer review is optimal.

    IDEA: What if law reviews accepted empirical articles through their regular process, with a policy that they will then be subsequently peer reviewed. After the author has a chance to fix any fixable problems raised by the reviewers, the remaining unaddressed peer comments would be published alongside the article. If the author is too embarrassed by those scathing comments, then he or she can withdraw. If not, the comments would give policymakers the timely guidance that Schwartz rightly says they need.

    Such a reform would probably be equally valuable for non-empirical articles. And at least that two people on earth will read the article.

  4. anon says:


    I very much agree with everything that you said! Well put.

    I prefer weak data to no data/rank speculation, and peer-review gatekeeping is not a panacea. Nice idea, although it would require people to step up to the plate and take on the task of post-publication peer review.

  5. Chris Griffin says:

    This is a very valuable post, David. A couple thoughts:

    1. “First, nearly all law reviews lack formal peer review. The lack of peer review potentially permits dubious data to be reported without differentiation alongside quality data.”

    Point taken, but this fact is quickly being diminished by the informal peer review that occurs at many of the law reviews in which empirical scholarship regularly appears. The student editors–even those with very good training in statistical social science–will seek feedback from an established ELS faculty member or a faculty member at the university’s economics department. Thus, although the faculty member is usually not asked to write a peer review report, there is some quality control that takes place before ELS work ends up in the pages of student-edited journals. ctr’s suggestion is a very good one, but student editors are already wary of publishing empirical work; a conditional acceptance in either direction could be too chaotic for them to manage.

    2. I like your preliminary proposal for how practitioners and scholars should deal with “weak data.” I imagine we all would prefer more scrupulous, cautionary reporting of empirical work. I don’t know that the situation is all that different from the peer-review domain, however. Let’s say that the universe of NBER Working Paper Series is effectively the same as ELS work that ends up in law reviews (very little to no peer review, subject to my point in #1). Econometric analysis enters the semi-public domain all the time, gets picked up by the Times and other outlets, and can be the subject of fierce debate for long periods (e.g., the Hoxby-Rothstein exchange; the recent Rogoff-Reinhart mess). My point is that the dangers are not limited to legal scholars who engage in empirical studies; our arts & sciences colleagues face the same incentive/obligation trade-off to get timely work to the public but do it honestly and carefully.

  6. ctr says:

    Just to clarify — my proposal is NOT that law reviews should conditionally accept empirical articles, subject to subsequent peer review. (Really that’s no acceptance at all.) Instead, I’m suggesting that the journals go ahead and accept the articles that they want to accept, with the condition that they will be peer reviewed, and after a chance for revision, the reviewers un-addressed comments will be published with the article. I suggested that the author could withdraw the article if they felt that the review was just too scathing, but maybe the publication commitment should be binding in both directions.