What Difference Representation: Clinical Trials
In What Difference Representation: Offers, Actual Use, and the Need for Randomization Jim Greiner and Cassandra Wolos Pattanayak present the results from a randomized controlled study that was designed to assess the efficacy of an offer of representation from the Harvard Legal Aid Bureau (HLAB)−a student-run provider of legal services that is part of the clinical education program at Harvard. There is a great deal to like about this article: it is methodologically rigorous, the data analysis is careful and transparent, its main points are clearly argued and explained, and both the specific results dealing with the efficacy of an offer of HLAB representation in unemployment benefits cases and the broader argument about the need for randomized controlled trials to better understand the effects of representation in civil proceedings are provocative and relevant. The authors are clear about what they are doing and what they are not doing and they offer good advice about how one might design additional studies to assess interesting questions that are outside the scope of their own study.
It would be easy to spend more time and space than this blog post permits to discuss everything that I like about this article. I am not going to do that.
Instead, I’d like to use this post to briefly mention some issues that arise when one attempts to seriously evaluate the efficacy of legal representation−particularly free legal assistance such as that offered by law school clinical education programs.That randomized controlled experiments, when ethically and practically possible, provide the best means of ascertaining the causal effect of some well-defined intervention on a particular outcome of interest is well accepted by statisticians and quantitative social scientists. Indeed Greiner & Pattanayak provide an accessible account of the many benefits of randomized treatment assignment. I completely agree with Greiner and Pattanayak about these benefits and the need for serious evaluation of programs that may have important effects on peoples lives.
Nonetheless, many providers of legal services are apprehensive about such studies. Random assignment of units to treatment and control means that researchers/clinicians have the ability to decide who is offered representation (active treatment) and who isn’t (control) and that they give up this ability in favor of random assignment. Real random assignment of treatment is more than just plausibly exogenous natural variation in treatment status−it involves a choice to curtail professional expertise and discretion in favor of a truly random decision rule and as Greiner & Pattanayak document, this makes many clinicians uncomfortable. As one noted, “We can see the blood now.”
Presumably this unease is at least in part due to some, possibly justified, belief that they know who will benefit from representation and who won’t. The double randomization scheme discussed by Greiner & Pattanayak is a clever way to assess whether particular providers can in fact accurately identify potential clients who will benefit from representation.
However, even before such an experiment can be conducted a series of ethical questions should be asked. Namely:
- Are we truly so unsure about the benefits of representation in the relevant context that randomly assigning offers of representation is ethically justified?
- If we have some, but not conclusive, knowledge of the effects of an offer of representation on some relevant outcomes should that information be conveyed to the experimental subjects (the clients)? If so, how?
- When should experimentation stop?
The results in Greiner & Pattanayak suggest that the answer to the first question is “yes” at least occasionally and perhaps substantially more often than many clinicians would have thought. This is an important point insofar as it clarifies what are, and are not, likely to be principled reasons for and against experimentation in contexts resembling their study.
On the second question, it seems to me that there is a case to be made for some sort of disclosure. Nonetheless, this is a much more complicated issue than it might appear because such disclosure would almost certainly diminish what could be learned from future studies and thus lead to more experimentation and greater exposure to less effective, and possibly harmful, interventions.
The question of when experimentation should stop might seem to have an easy answer−it should stop when the relevant effects are well enough understood to guide future decision making. The problem is that, unlike most medical studies that deal with a slowly changing population in a basically static context, legal proceedings take place among forward-looking actors who can rapidly adapt to changes made by other actors. In short, there may be general equilibrium effects that make seemingly conclusive experimental evidence less relevant than we might think.
In addition, there is a very difficult methodological question regarding what counts as evidence about the effects of representation. In my mind, this is the central question. Some might argue that the only principled sources of evidence are the data from a randomized controlled experiment or experiments. While I agree that such data would constitute very strong evidence I don’t think experimental evidence is always necessary. For instance, one does not need a randomized controlled trial to believe that parachutes often prevent death from high altitude falls. The effect of legal representation in capital cases may not be quite that obvious but I think most people would agree that it is unlikely to have a negative effect and, at least in some cases, is likely to have a positive effect. There is probably somewhat less agreement as to whether good clinicians in this area can spot the cases where they can make a positive effect but if they can then there is no easy case to be made for randomly assigning offers of representation to capital defendants−even with no prior experimentation.
The effects of representation are rarely this clear in the civil cases considered by Greiner & Pattanayak. Nonetheless, criteria for determining what counts as legitimate evidence of effects are still necessary if one is to consider the ethical implications of proposed randomized studies.
The question of who benefits from legal representation is extremely important and Greiner & Pattanayak have done a great service by showing us that the answer to this question is much more complicated than many of us thought. Serious evaluation of legal aid programs requires more than just anecdotal accounts of success and simple observational, i.e. non-randomized, studies. That said, randomized experiments have their own unique ethical and epistemological issues that researchers need to take seriously. Moving forward with principled evaluation of legal aid programs is something that requires input from clinicians, empiricists schooled in methods of program evaluation, and ethicists. If such collaboration happens we (including those in need of legal aid) will all be better off.