In a draft essay, Service Delivery, Resource Allocation and Access to Justice: Greiner and Pattanayak and the Research Imperative, Tony Alfieri, Jeanne Charn, Steve Wizner, and I reflect on Jim Greiner and Cassandra Pattanayak’s provocative article reporting the results of a randomized controlled trial evaluating legal assistance to low-income clients at the Harvard Legal Aid Bureau. (The Greiner and Pattanayak article was the subject of a Concurring Opinions symposium last March.) Studying the outcomes of appeals from initial denials of unemployment insurance benefit claims, Greiner and Pattanayak asked, what difference does legal representation make? Their answer is that “an offer of HLAB representation had no statistically significant effect on the probability that a claimant would prevail, but that the offer did delay the adjudicatory process.” That is, not only was an offer of legal assistance immaterial to the case outcome, it may have harmed clients’ interests. The Greiner and Pattanayak findings challenge our intuition, experience and deeply-held professional belief that lawyer representation of indigent clients in civil matters is fundamental to the pursuit of justice. Our first reaction is that the study must have fatal conceptual or methodological flaws – the researchers studied the wrong thing in the wrong way. Even when we learn that the study is credible and well designed, we doubt that this kind of research is a worthwhile use of our time or money relative to serving needy clients. Finally, and perhaps most importantly, we worry that the published results will only serve as fodder for the decades-long political assault on legal services for the poor. If replicated across venues, however, studies like Greiner and Pattanayak’s can tell us a great deal about individual representation, program design and systemic access to justice questions. In fact, we cannot make genuine progress in any of these areas – much less marshal the case for more robust legal aid investments and the right to counsel in some civil cases – without better evidence of when, where and for whom representation makes a difference. Fortunately, developments in law schools, the professions and a growing demand for evidence-driven policymaking provide support, infrastructure and incentive for such research. For these reasons, we urge legal services lawyers and clinical law professors to collaborate in an expansive, empirical research agenda.
Category: Symposium (What Difference Representation)
Thanks to all our participants for an amazing symposium. You can see all twenty-two (!) posts here. Thanks especially to Jim and Cassandra for being fantastically good sports in subjecting their work to public scrutiny and to replying so conscientiously to everyone’s comments. Richard Zorza put it best: “[T]his study, regardless of, or perhaps because of, its controversial nature, will be looked back at as a critical event in the history of access to justice.” If he’s right, I really proud that CoOp could serve as a platform to debate the article, and the shape of things to come.
I write as a legal services lawyer and clinician who practiced in a neighborhood general law practice for low and moderate income people for over three decades and for twenty seven years directed a large clinical practice site at Harvard Law School. The issue of case selection/triage, to which this paper is relevant, was always a challenge. We had inklings that we needed more rigorous approaches and even had a socical scientist on staff for a few years but were never able to conceive and carry out a serious study, let alone one as sophisticated as the G&P randomized trial. I whole-heartedly welcome this effort and the the authors challenge to engage in rigorous scrutiny of the actual workings of legal services delivery in the U. S. Serious empirical work goes on as a matter of course in peer nations, all of which have been successful in obtaining and holding substantially greater resources than in the U. S. I don’t assume a connection between the research programs and better funding (hasn’t been studied) but these programs know a lot more about what they are producing and have refined delivery approaches and policies based on what they have learned. Regarding the posts expressing concern that data and studies may be used improperly. I agree with Richard Zorza that we can’t be sure this won’t occur , but I believe the risks are much greater if we continue with virtually no serious effort to collect good outcome data, comparatively study different approahces to service delivery, develop productivity and efficiency standards as well as good measures of quality . Because we don’t have even a decent data system, we cannot assure that we are making the best use of the resources available. What if thousands more people could be effectively helped if programs were more efficient, targeted resources more effectively, and leveraged expertise to maximize both cost and outcome effectiveness? The result would be the same as if we had substantially more resources. Having located myself firmly in the “we need more of just this sort of high quality research,” here are some thoughts, in no particular order, about the value – I would say the necessity – of a bold, broad empirical scrutiny of “our fondest pet notions” about our work. As I edit, I see the post is getting long, but I teach until 9 pm tonight, so may just have time to get this in before the symposium closes! I am encouraged by evidence that claimants succeed via self-representation or with information or limited advice and assistance. Advocacy resources can be directed to matters where advice/self-help is sub-optimal. I remain attached to the early goals of client activation and empowerment and self-help may play a role – a possiblity for further study! I understand the offer/representation distinction in the study and recognize that win/lose at hearing is not the only measure of success but a study should be assessed on what it purports to measure not on everything it could have measured. I support a broad and long term research agenda recognizing that good studies help us frame issues for further study. The issue of quality of work by HLAB students comes up in...
A few suggested additions to Richard Zorza’s proposed “best practices” for randomized study of legal services: (1) Remember the distinction between what is measurable and what is important. Broadly speaking, legal services programs are trying to increase access to justice (make voices heard), solve individual or group problems (win cases), and change the legal environment so that poor people’s lives will be better (change laws or systems). Our work on these efforts is interwoven; for example, we take individual cases that will both provide access to the system and fix a person’s problem, and in that process we gain important information about what may be broken in the larger system and where solutions may lie. So: a large volume practice in an area of law that has a stream of comparable cases can be studied through randomization. On the other hand, efforts to change laws or systems, and innovative start-up projects, must be evaluated through other means. The corollary here is: Be willing to publicly state, “This is a type of work that is susceptible to this research tool; there are other valuable types of work that must be studied with other tools.” (2) Be clear about what is (and isn’t) being studied. This may be a warning primarily aimed at the legal aid providers. Over time, we will want to learn how much impact our scarce resources can have in various areas of legal work in different jurisdictions for clients with different fact patterns, personal skills, age, linguistic abilities, mental health or physical characteristics using a variety of different intervention levels and strategies (i.e. advice vs. limited representation vs. long-term representation) and employing a variety of different personal advocacy skills (i.e. confrontational vs. compromising, high-level listening skills vs. high level speaking skills). We will need patience and persistence. Over time our services will be enhanced by exploring all of these questions (and more!). But we will get garbage results if we try to do everything at once. The Greiner and Pattanayak HLAB study, and all the commentary in this symposium, illuminates how much work we have to do. Did the Harvard students have no impact (one commentator disagrees based on the data). Could a change in client selection enhance the impact for the clients served? A change in case strategy? A change in law student advocacy style or skills? We are so early in this learning process that for now, each study will primarily highlight the next set of questions to be asked. (3) Be aware of the costs of measurement. Measurement takes time. When we say “legal aid to the poor is a scarce resource,” we mean that there are nowhere near enough people-hours to do all that we know justice requires. Planning and carrying out a useful measurement (a “next step” in the learning process described above) takes time away from other activities. We will have to think through, design, set up the study. We will have to explain to staff and communities we serve and funders...
Jim and Cassandra write: “To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers. We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings. We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive. Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful.” I meant to cabin my argument to law school clinics. And I do understand that there may be very rare cases where collecting outcomes will hurt clients (such as deportation). But what about a clinic that focuses on “systemic change.” Let’s assume that subsidizing such a clinic would be a good thing for a law school to do (or, put it another way, we think it is a good idea for current law students to incur more debt so that society gets the benefit of the clinics’ social agitation). Obviously, randomization of client outcomes would be a terrible fit for measuring the success of such a clinic. It would be precisely the kind of lamppost/data problem that Brian Leiter thinks characterizes much empirical work. But that doesn’t mean that randomization couldn’t be useful in measuring other kinds of clinic outcomes. What about randomization in the allocation of law student “employees” to the clinic as a way to measure student satisfaction in the “learning outcomes“? Or randomization of intake and utilizing different client contact techniques as a way of measuring client satisfaction with their representation (or feelings about the legitimacy of the system?) One thing that the commentators in this symposium have tried to emphasize is that winning & losing aren’t the only outputs of the market for indigent legal services. Controlled study of the actors in the system needn’t be constrained in the way that Jim and Cassandra’s reply to my modest proposal to mandate randomization suggest.
Thanks to Jim and Cassandra for their carefully constructed study of the impact of an offer from the Harvard Legal Aid Bureau for representation before the Massachusetts Division of Unemployment Assistance, and to all of the participants in the symposium for their thoughtful contributions. What Difference Representation? continues to provoke much thought, and as others have noted, will have a great impact on the access to justice debate. I’d like to focus on the last question posed in the paper — where do we go from here? — and tie this in with questions about triage raised by Richard Zorza and questions about intake processes raised by Margaret Monsell. The discussion below is informed by my experience as a legal service provider in the asylum system, a legal arena that the authors note is strikingly different from the unemployment benefits appeals process described in the article. My first point is that intake processes vary significantly between different service providers offering representation in similar and different areas of the law. In my experience selecting cases for the asylum clinics at Georgetown and Yale, for example, we declined only cases that were frivolous, and at least some intake folks (yours truly included) preferred to select the more difficult cases, believing that high-quality student representation could make the most difference in these cases. Surely other legal services providers select for the cases that are most likely to win, under different theories about the most effective use of resources. WDR does not discuss which approach HLAB takes in normal practice (that is, outside the randomization study). On page twenty, the study states that information on financial eligibility and “certain additional facts regarding the caller and the case” are put to the vote of HLAB’s intake committee. On what grounds does this committee vote to accept or reject a case? In other words, does HLAB normally seek the hard cases, the more straightforward cases, some combination, or does it not take the merits into account at all?
We thank Kevin Quinn and David Hoffman for taking the time to comment in our paper. Again, these are two authors whose work we have read and admired in the past. Both Dave and Kevin offer thoughts about the levelof enthusiasm legal empiricists, legal services providers, and clinicians should have for randomized studies. We find ourselves in much but not total agreement with both. To Kevin, we suggest that there is more at stake than just finding out whether legal assistance helps potential clients. In an era of scarce legal resources, providers and funders have to make allocation decisions across legal practice areas (i.e., should we fund representation for SSI/SSDI appeals or for unemployment appeals or for summary eviction defense). That requires more precise knowledge about how large representation (offer or actual use) effects are, how much bang for the buck. Perhaps even more importantly, scarcity requires that we learn how to triage well; see Richard Zorza’s posts here and the numerous entries in his own blog on this subject. That means studying the effects of limited interventions. Randomized trials provide critical information on these questions, even if one agrees (as we do) that in some settings, asking whether representation (offer or actual use) helps clients is like asking whether parachutes are useful. Thus, perhaps the parachute analogy is inapt, or better, it requires clarification: we are in a world in which not all who could benefit from full-service parachutes can receive them. Some will have to be provided with rickety parachutes, and some with little more than large blankets. We all should try to change this situation as much as possible (thus the fervent hope we expressed in the paper that funding for legal services be increased). But the oversubscription problem is simply enormous. When there isn’t enough to go around, we need to know what we need to know to allocate well. Meanwhile, randomized studies can also provide critical information on the pro se accessibility of an adjudicatory system, which can lay the groundwork for reform. To Dave, we say that our enthusiasm for randomized studies is high, but perhaps not high enough to consider a duty to randomize among law school clinics or among legal services providers. We provided an example in the paper of practice in which randomization was inappropriate because collecting outcomes might have exposed study subjects to deportation proceedings. We also highlighted in the paper that in the case of a practice (including possibly a law school clinic) that focuses principally on systemic change, randomization of that practice is not constructive. Instead, what should be done is a series of randomized studies of an alternative service provider’s practice in that same adjudicatory system; these alternative provider studies can help to assess whether the first provider’s efforts at systemic change have been successful. Our great thanks to both Kevin and Dave for writing, and (obviously) to Dave (and Jaya) for organizing this symposium.
In What Difference Representation: Offers, Actual Use, and the Need for Randomization Jim Greiner and Cassandra Wolos Pattanayak present the results from a randomized controlled study that was designed to assess the efficacy of an offer of representation from the Harvard Legal Aid Bureau (HLAB)−a student-run provider of legal services that is part of the clinical education program at Harvard. There is a great deal to like about this article: it is methodologically rigorous, the data analysis is careful and transparent, its main points are clearly argued and explained, and both the specific results dealing with the efficacy of an offer of HLAB representation in unemployment benefits cases and the broader argument about the need for randomized controlled trials to better understand the effects of representation in civil proceedings are provocative and relevant. The authors are clear about what they are doing and what they are not doing and they offer good advice about how one might design additional studies to assess interesting questions that are outside the scope of their own study. It would be easy to spend more time and space than this blog post permits to discuss everything that I like about this article. I am not going to do that. Instead, I’d like to use this post to briefly mention some issues that arise when one attempts to seriously evaluate the efficacy of legal representation−particularly free legal assistance such as that offered by law school clinical education programs.
Lurking behind much of the debate in this symposium is anxiety that negative findings about access to justice services will strengthen and facilitate attempts to reduce resources for access to justice services. While it would be impossible to rebut the claim that this might happen, that can not be an argument against conducting or reporting research. On the contrary, it has to be an argument for more and better research. Having been involved in access to justice for decades, I am all too aware that it never seems to be the right time to make ourselves vulnerable — so the answer has to be always, because that is the only way to gain credibilit.. But this does all raise the question as to how research should be structured, analyzed, and particulalry, reported in order to minimize the risk of results inconsistent with the ultimate findings of the research, in all their complexity and subtlety. Some thoughts: Generality of Reporting. The headings, abstract, etc, must be structured to accurately reflect the generlity of the research The Reporting of Context. Studies should be very careful about describing accurately the context in which treatment is provided. Randomization/Observation. Where studies are not randomized, that must be very clearly reported, and selection bias must be loudly proclaimed, not the subject of a footnote. Explanation of Statistical Significance. The issues of statistical significance must, to the extent possible, be explained as clearly as possible, in lay terms. The failure to do so, when it occurs, makes both overstatement an unfair critique easier. Lay Version. Research should be made available in a lay summary version, without complexities but with the detail and cautions. This will reduce the risk of the results being oversimplified by the media and/or others Vigilance as to Over/Under Generalization. The text should not only be accurate as to the level of generality of the research, but should be explicit as to the kinds of generalizations that might erroneously be drawn from the research. (This makes it easier to rebut overdrawn conclusions made by legal or political opponents.) I would very much appreciate additions to such best practices. There are surely many more in the social sciences. Those suggested here, however, are less about avoiding error, and more about avoiding error in the reporting or use by others.
Thanks for the invitation to participate in this interesting and provocative symposium. I’m a legal services attorney in Boston. My employer, Massachusetts Law Reform Institute (MLRI), has as one of its primary tasks to connect the state’s field programs, where individual client representation occurs, with larger political bodies, including legislatures and administrative agencies, where the systemic changes affecting our clients most often take place. (The legal services programs in many states include organizations comparable to MLRI; we are sometimes known by the somewhat infelicitous name “backup centers.”) Among the programs with which MLRI is in communication is the Harvard Legal Assistance Bureau, and I would take this moment to acknowledge the high regard in which I and my colleagues regard their work. The substantive area of my work is employment law. It is no surprise that during the past three years of our country’s Great Recession, the importance of the unemployment insurance system for our clients has increased enormously and, consequently, it has occupied a greater portion of my time than might otherwise have been the case. I’m not a statistician nor do I work in a field program representing individual clients, so my comments will not address in any detail the validity of the HLAB study or the conclusions that may properly be drawn from it. As one member of the community of Massachusetts legal services attorneys, however, I have an obvious interest in the way the study portrays us: we are variously described as self-protective, emotional, distrustful of being evaluated, and reluctant to the point of perverseness in participating in randomized studies of the kind the authors wish to conduct. Our resistance in this regard has itself already been the subject of comment here. Happily, it is not often that one looks into what seems to be a mirror and sees the personage looking back wearing a black hat and a snarl. But when it does happen, it’s hard to look away without some effort at clarification. So I will devote my contribution to the symposium to the topic of the perceived reluctance of the legal services community to cooperate in randomized trials. It goes without saying, but the following thoughts are those of only one member of a larger community. My understanding is that in the HLAB study, no significant case evaluation occurred prior to randomization. Many of us in legal services view with trepidation the idea of ceding control over case selection to the randomization process. Others have more sanguine views, either because they assume that randomization is already taking place or that it ought to be. For example, in his comments from a few months ago, Dave Hoffman was working under the assumption that to randomize client selection would not change an agency’s representation practices at all, and on that basis, he criticized resistance to randomized control trials as “trying to prevent research from happening.” The authors of the study are enthusiastic about randomization not only because of its scientific value in statistical research but...