Should Empirical Legal Scholars Have Special Responsibilities?

You may also like...

6 Responses

  1. anon says:

    I think that the best solution is not to publish empirical research in student-edited, non-peer reviewed journals. It is not reasonable to assume that law students lacking empirical training will be able to discern the quality of an empirical study. Inevitably, they will pick articles with sexy topics or conclusions, regardless of the strength of the data.

  2. TJ says:


    I’m going to be less generous than you and say that, in many instances where I see what you describe as misuse of weak data by others, the misuse is likely intended by the author. That is, the most salient example of this problem to me is when an author reports the weak data, makes a bunch of policy-salient inferences prominently in the abstract and introduction, and then somewhere in the middle lists all the limitations and alternative inferences that can be drawn. If this kind of intentional sleight-of-hand is the core example of the problem, I don’t think it is a “special” obligation of an empirical researcher to refrain from it.

  3. ctr says:

    Most of the analysis by Schwartz and Anon seems equally applicable to non-empirical research. I see hasty generalizations and weak arguments under-determined by the available evidence in all sorts of scholarship.

    The distinguishing factor that Schwartz cites — that empirical research may have a disproportionate impact — is interesting. Perhaps, ceteris parabis, that’s an argument for more empirical scholarship, assuming that its not junk on the median.

    It bears emphasis that the alternative to weak data is often rank speculation. The courts and policymakers (and scholars) ROUTINELY rely on raw assumptions that align with their policy preferences or status quo biases. Even weak data — an accumulation of anecdotes — can light a candle to the darkness. Considering your 6/10 example — that finding could have real epistemic value if the alternative hypothesis is that 0% or 10% of the cases do X. (Assuming random sampling, etc.) Validity and statistical significance are relative concepts, and we are in a world of scarcity.

    Finally, anyone who has published in peer reviewed journals can attest to the large variance in the quality of peer reviewers. Frankly, the process suffers from a huge sampling problem itself — one or two hand-picked reviewers purport to speak for the rigor and other values of the entire field of hundreds of scholars, and thus get to be the secret censors? (Want to figure a p-value on that?)

    I have seen peer reviewers of empirical papers that couldn’t help but spend page after page arguing about the value-laden and political implications of the work, and with those glasses firmly affixed trying to also evaluate the scientific rigor of the study. This is just to say that we are all human; I may be guilty of the same.

    Even if the two censors are representative of the field, that process causes many false positives and false negatives too, errors which are not random but are biased towards paradigm-maintenance. Perhaps a politically-charged field like ours is not one where censorship by peer review is optimal.

    IDEA: What if law reviews accepted empirical articles through their regular process, with a policy that they will then be subsequently peer reviewed. After the author has a chance to fix any fixable problems raised by the reviewers, the remaining unaddressed peer comments would be published alongside the article. If the author is too embarrassed by those scathing comments, then he or she can withdraw. If not, the comments would give policymakers the timely guidance that Schwartz rightly says they need.

    Such a reform would probably be equally valuable for non-empirical articles. And at least that two people on earth will read the article.

  4. anon says:


    I very much agree with everything that you said! Well put.

    I prefer weak data to no data/rank speculation, and peer-review gatekeeping is not a panacea. Nice idea, although it would require people to step up to the plate and take on the task of post-publication peer review.

  5. Chris Griffin says:

    This is a very valuable post, David. A couple thoughts:

    1. “First, nearly all law reviews lack formal peer review. The lack of peer review potentially permits dubious data to be reported without differentiation alongside quality data.”

    Point taken, but this fact is quickly being diminished by the informal peer review that occurs at many of the law reviews in which empirical scholarship regularly appears. The student editors–even those with very good training in statistical social science–will seek feedback from an established ELS faculty member or a faculty member at the university’s economics department. Thus, although the faculty member is usually not asked to write a peer review report, there is some quality control that takes place before ELS work ends up in the pages of student-edited journals. ctr’s suggestion is a very good one, but student editors are already wary of publishing empirical work; a conditional acceptance in either direction could be too chaotic for them to manage.

    2. I like your preliminary proposal for how practitioners and scholars should deal with “weak data.” I imagine we all would prefer more scrupulous, cautionary reporting of empirical work. I don’t know that the situation is all that different from the peer-review domain, however. Let’s say that the universe of NBER Working Paper Series is effectively the same as ELS work that ends up in law reviews (very little to no peer review, subject to my point in #1). Econometric analysis enters the semi-public domain all the time, gets picked up by the Times and other outlets, and can be the subject of fierce debate for long periods (e.g., the Hoxby-Rothstein exchange; the recent Rogoff-Reinhart mess). My point is that the dangers are not limited to legal scholars who engage in empirical studies; our arts & sciences colleagues face the same incentive/obligation trade-off to get timely work to the public but do it honestly and carefully.

  6. ctr says:

    Just to clarify — my proposal is NOT that law reviews should conditionally accept empirical articles, subject to subsequent peer review. (Really that’s no acceptance at all.) Instead, I’m suggesting that the journals go ahead and accept the articles that they want to accept, with the condition that they will be peer reviewed, and after a chance for revision, the reviewers un-addressed comments will be published with the article. I suggested that the author could withdraw the article if they felt that the review was just too scathing, but maybe the publication commitment should be binding in both directions.