Good to Evaluate Our Pet Notions
I’m delighted that Jim and Cassandra have conducted this study. We have accomplished many things in clinics, but we have not even scratched the surface of their potential to serve as sites of inquiry – what some of us have called “clinic lab offices” – to ask basic service delivery questions. Their literature review is a cautionary tale about what we know (or think we know) about the impact of representation. Their original research is pathbreaking, if controversial.
There is plenty to praise here, which I have done privately to the authors, but like Kevin I will focus on some of the challenges. In addition to the methodological limits of this study, the paper raises a lot of important questions that can only be answered through additional research, especially with respect to causal mechanisms. I’ve shared specific questions and suggestions in recent months, but my general view is that an organizational sociology/ethnography dimension – to generate more data about the administrative agency, the service provider and this particular subset of clients – would inform this study (and others) to help account for otherwise not obvious variables.
For example, w/r/t to the administrative agency, what incentives exist for ALJ to grant/deny benefits, especially to represented vs. unrepresented claimants? Do ALJs get punished/rewarded for false positives/negatives? My experience with agency adjudication is that there are often hidden nudges that operate much more powerfully than surface considerations (Social Security funds district offices based on the handling of some kinds of claims but not others; VA rewards certain workers for productivity, but not accuracy, etc.). How has this venue changed over time in response to focused representation by HLAB? Maybe this is a story of triumph – HLAB representation has been so effective that unrepresented claimants now have a level playing field, or at least a user-friendly setting.
In empirical work, the best data (in statistically significant terms) is often generated by the most narrowly drawn question (by eliminating/controlling for other causal mechanisms). That is, we sacrifice breadth for depth, which is a version of the limits that have frustrated critics of this study. RCTs are the research gold standard, but socially complex relationships can rarely be reduced to quantitative assessments, and are often better understood through the application of mixed methods, including a thick qualitative description of what’s motivating various actors within the system.
In 1967, referring to the HLS Community Legal Assistance Office established a year earlier and funded as a demonstration project by the federal Office of Economic Opportunity, Frank Michelman observed: “[W]e have undertaken to construct and demonstrate what we have been pleased to call a ‘model’ of a law-school-affiliated legal-services program. Stripped of pretension and reduced to practicality, what this means to me is that we are committed to a continuing effort to generate alternative methods, to put into operation whatever recommends itself to our objective appraisal, and to evaluate remorselessly our fondest pet notions.”
It’s taken more than 40 years, but Jim and Cassandra’s paper, with all its strengths and limits, is a testament to the potential and pitfalls of evaluating remorselessly our fondest pet notions. Hopefully the interest it has generated – and their forthcoming work, which I eagerly await – will spur others to undertake rigorous study of legal representation on behalf of the poor. The absence of such data impairs our ability both to allocate scarce resources effectively and to make the case persuasively to public and private funders for greater investments in the field.