Setting the Bar, and the Limits of Empirical Research

Larry Ribstein and Jonathan Wilson are debating the merits of a strong, exclusionary, state bar.

Wilson’s position is pro-Bar:

Deregulating lawyers as punishment or retribution for a profession that has lost its way would be a recipe for disaster. Deregulating the practice of law would open the floodgates to fraud of every conceivable variety and would only compound the problems that the readers of these pages see in our civil justice system.

Ribstein, naturally, is pro-market:

Big law firms provide a strong reputational “bond” . . . Lawyers can be certified by private organizations, including existing bar associations, which can compete with each other by earning reputations for reliability. . . .We could have stricter pleading rules, or require losers to pay winners’ fees. Or how about this: let anybody into court, but adopt a loser pays rule for parties that come into court represented by anything less than a lawyer with the highest possible trial certificate . . . Even if only licensing would effectively deal with this problem, the licensing scheme should be designed specifically to protect the courts. Instead of requiring the same all-purpose license to handle a real estate transaction and to prosecute a billion-dollar class action, we could have a special licensing law for courtroom practice, backed by tight regulation of trial lawyers’ conduct – something like the traditional barrister/solicitor distinction in the UK.

Josh Wright has picked up the thread of the discussion at TOTM, and suggests that empirical evidence would inform this debate. Unfortunately, as both Larry and he note, there is a paucity of useful studies on point:

If I recall, the Federal Trade Commission has recently been involved in some advocacy efforts in favor of limiting the scope of unauthorized practice of law statutes. My sense is that a number of states must have relaxed unauthorized practice of law restrictions (I think Arizona is one), or similarly relaxed restrictions on lawyer licensing, such that one could directly test the impact of these restrictions on consumers in terms of prices and quality of service. There must be work on this somewhere.

Solove and I have gone around on this question before (see here for the powerful pro-licensing position, and here and here for Solove’s “response”).

Generally, I like Josh’s intuition. It would be quite useful to look to Arizona, or other natural experiments, to help us to answer the problem of the utility of the Bar Exam and other licensing barriers. Surely, there is no reason in the abstract to preserve an ancient system that keeps lawyer fees artificially high, diverts millions of dollars from law students to Barbri, and causes no end of mental anguish simply because it provides a new jurisprudential lens!

But I’m quite skeptical that this is an answerable question, at least in the short term. My thinking is informed somewhat by the new Malcolm Gladwell New Yorker essay about basketball. Although Gladwell extols the virtues of statistical analysis (instead of anecdote, judgment, and valuing the joy of watching Allen Iverson triumph despite his height), the lesson I took from the piece was that:

Most tasks that professionals perform . . . are surprisingly hard to evaluate. Suppose that we wanted to measure something in the real world, like the relative skill of New York City’s heart surgeons. One obvious way would be to compare the mortality rates of the patients on whom they operate—except that substandard care isn’t necessarily fatal, so a more accurate measure might be how quickly patients get better or how few complications they have after surgery. But recovery time is a function as well of how a patient is treated in the intensive-care unit, which reflects the capabilities not just of the doctor but of the nurses in the I.C.U. So now we have to adjust for nurse quality in our assessment of surgeon quality. We’d also better adjust for how sick the patients were in the first place, and since well-regarded surgeons often treat the most difficult cases, the best surgeons might well have the poorest patient recovery rates. In order to measure something you thought was fairly straightforward, you really have to take into account a series of things that aren’t so straightforward.

I know how I would test the direct cost of legal service in Pennsylvania, and I’ve no doubt that it would go down if I (by fiat) abolished the state bar. But I have no good idea of how we can measure lawyer “quality”. To take something as obvious as criminal defense, some really good public defenders will lose every case for a year, but take comfort in having not lost on the top count of a single indictment. Saying that a public defender who went 0 for 50 in 2005 was a less “good” attorney than a prosecutor who went 50-0 would be a real problem. Facts drive litigation, and make empirical investigation of lawyer quality as a quantitative matter hard. And that is for attorneys who perform in public. How do you evaluate the relative strength of deal counsel on a gross level? Count the typos in the document? Talk with the business folks, and ask who got in the way less? [Obviously, deal counsel can be very good and very bad: the point is we need metrics that are easily coded by, say, research assistants.]

So here is the question for our readers. Can you design an empirical project that measures both litigation and transactional practice quality as a function of licensing?

You may also like...

8 Responses

  1. John Armstrong says:

    My big problem with Ribstein’s position is where he states

    Lawyers can be certified by private organizations, including existing bar associations, which can compete with each other by earning reputations for reliability

    So, if I read this correctly he’s saying there would be a number of independent groups willing to attest to the quality of lawyers, and they would gain reputations for reliability and trustworthiness in their assessments. I’m sure they would gain such reputations within the legal community, but I have a feeling quite a large number of people pick a lawyer from outside that community, and in times of rather dire need and stress. As it stands, any practicing lawyer is attested to have some basic level of competance, and a distraught prospective client can at least have that assurance.

  2. Seth R. says:

    I don’t have time to comment on the entire post. But one comment on Ribstein’s seeming proposition that deregulation would strike a blow against biglaw …

    Biglaw is exactly the segment of the Bar that would be least affected by deregulation.

    Can you imagine trying to practice family law as a solo practitioner with the added problem of having to reduce your fees to compete with all the local bargain basement offerings?

    Family lawyers already don’t make a lot of money and struggle to stay in business. Turning the whole practice area into glorified paralegal work would absolutely annihilate the family law solo practitioner. The work would be taken over instead, by larger commercial “divorce mills.” Same with immigration law, same with bankruptcy law, same with criminal law.

    I guess whether you prefer talking to your friend’s personal lawyer about your marital problems over calling 1-800-DIVORCE is a matter of personal preference.

    But biglaw would be largely immune to this. They’re clients tend to have the financial resources to pay for “the best.” Exxon Mobile isn’t about to drop Holland and Hart for representation of its Wyoming oil interests simply because Congress decides to “deregulate the bar.”

    But plenty of Main Street Americans would drop their local solo bankruptcy attorney in favor of a cheaper online “debt counselor” financed by credit card companies.

    As usual, calls for deregulation take aim at the fat cats with a bazooka, miss entirely, and annihilate the small bystanders.

  3. Ted says:

    Seth’s comment is effectively saying that deregulation won’t affect those who have specialized skills in high demand, while will hurt those who are able to charge above-market rates to consumers without offering anything special that a paralegal couldn’t do. I fail to see why that’s a bug, rather than a feature.

  4. Dave, you are surely right that measures of quality will be imperfect. But I am not quite willing to surrender the utility of empirical analysis here. One might imagine that consumer complaints or formal disciplinary actions might be relevant measures, malpractice insurance rates, and there could certainly be others. I recall a paper by Muris and McChesney which used (among other things) direct measures such as dollar amounts in child support awards (they were comparing lawyers who did and did not advertise). My point is not to say that any of these measures, and I am sure they are others which are superior to those I mention here as examples, are perfect. To the extent that your point was that empirical analysis of quality is tough, Im on board.

    But, building on the some of the earlier comments and Wilson’s theory that it is the individual consumer will be hurt, perhaps it would be useful to look at transactions like real estate closings, divorce settlements, etc. These are the types of transactions where it is theorized consumers of legal services will be harmed. I believe a study, even one using imperfect quality measures, exploiting the natural experiments in UPL laws would provide some useful meat for policy analysis.

    I should also be clear that I have not completely surveyed the literature on the relationship between licensure and quality, but a survey I came across in an Enyclopedia of Law and Economics, at states the following: “Most of the evidence on this issue, however, suggests that licensing has, at best, a neutral effect on quality and may even harm consumers.”

  5. Dan Spevak says:

    What is up with Dave Hoffman’s insistance to measure everything scientifically or empirically? Some things don’t lend themselves to being measured, as the example with the surgeon illustrates: there are too many factors and variables. Same for lawyer “quality.” What exactly is that? Isn’t that sort of like an empirical measurement of beauty? Well, Anjelina Jolie has full sexy lips which we all would love to kiss, but Jennifer Anistor has that personableness to which, at the end of the day, or at the end of the sex, is would be just so great to relax next to… Some consumers value cheap lawyers, and for those lawyers offering lowest fees are of best quality; other consumers want personal attention, so lawyers who return calls regularly are best; yet other consumers want to see their enemies bleed, so most vicious lawyers are best. Eye of the beholder, anyone? Measuring quality involves making the choice about what is valued, and the components going into that choice will dictate your results, even if you had the perfect ’emprical test tube’ for gathering and analyzing the relevant data. The whole point is that you cannot delineate the scope of relevant factors without sacrificing objectivity, empiricism, and scientificism, so pleez, let’s not pretend that the exercise is anything but a change in form without any change in substance. To change substance, the only experiment that will tell you what consumers truly prefer, and what the market truly will bear is a wholesome deregulation of every hoop that one must jump through for the “honor” to practice law. This includes not only getting rid of the bar, but getting rid of law schools as prerequisites for entering the profession. My 8-year old son, frankly, can copy and paste the contracts clauses I spend 14 hours a day copying and pasting, and I’d really he rather not spend money on 3 years of bs and torture, crowned by the thorn to the profession that is the bar exam. I wonder if there is even a single person out there who would willingly submit that the bar exam was personally useful, and a good measure of legal skill or promise.

    You want an experiment? Here’s one: ask practicing attorneys what requirements imposed on them they found valuable in the practice of law. Please include those lawyers who started off not at law schools, but as apprentices. Also, ask every consumer of family law whether they’d be willing to forego the “quality controls” of the regulated bar in favor of actual competition and yes, clinics staffed by non-attorneys who don’t have to charge prices high enough to recoup the absurd costs of legal education. As for the comment regarding the decrease in quality, and the fear that ‘debt attorneys will be bought up by credit card companies,’ any practitioner who knows anything about consumer protection laws knows this will not happen so long as there are fee shifting statutes allowing the plaintiffs’ attorneys to retain their independence. You want quality for the poor? Write to your congressman and urge more fee shifting statutes passed in a variety of other contexts. Or, you could write a fee shifting statute onto a legal malpractice, how’s that for direct regulation of quality?

    As for family law, frankly, if it can be done cheaper (by improving competition and abolishing the monopoly on lawyers doing dumb paperwork), then it should be done cheaper. I am sickened by the snobbery of our profession, perhaps even more then by our pretenses. If I could rule by fiat, I’d ban the cartel that our profession is run by.

  6. Dave Hoffman says:

    Josh says: “consumer complaints or formal disciplinary actions might be relevant measures, malpractice insurance rates, and there could certainly be others. I recall a paper by Muris and McChesney which used (among other things) direct measures such as dollar amounts in child support awards (they were comparing lawyers who did and did not advertise).”

    I know you don’t mean to be exclusive, but all of these measures raise immediate objections.

    Consumer complaints and disciplinary actions are extremely weak proxies for quality. In the current system, the likelihood of such grassroots generated action is vanishingly small. Unlike, say, citizen review boards over police departments, the obscurity and timing problems of legal malpractice make it unlikely that complaints are going to be sufficiently nonrandom to measure much besides lawyer personality.

    Malpractice rates as a quality proxy have been tried with respect to doctors, and the sense I have from the literature is that insurers don’t spend the money monitoring and thus do not target rates well to care levels.

    Child support awards – like win rates in criminal trials – are useful (if at all, see post) for those small set of lawyers who litigate in court.

    I agree with Josh that this isn’t an impossible project. But I think I’m more skeptical than he that it is solvable in the near future. Thus, Dan Spevak, I think you’ve misread me.

  7. Dave, I meant neither to be exclusive, nor necessarily endorse or criticize these measures. I certainly agree with you that they are imperfect. That said, any empirical evidence would be a welcome addition and represent an improvement over hand waving and mere speculation as to future effects, a point upon which I think we both agree.

  8. shamu says:

    can one design an empirical project that measures both litigation and transactional practice quality as a function of licensing?

    well, i suppose that in addition to the numerical measures already suggested (verdicts, malpractice judgments, etc.), one could conduct exit surveys of consumers of a particluar legal service (e.g., divorce or bankruptcy), to determine reasons, expectations, and satisfaction levels for a client’s selecting of and experience with a legal services provider (lawyer or non). this would essentially be a qualitative measure, and may capture the one quality-control measure currently operating in the legal market: “reputation,” a measure on which the us news law school rankings heavily depend, and on which chambers & partners has ranked lawyers & firms. perhaps research as to the quality of non-licensed entities would lead to an exception to the bar exam, which i agree creates unnecessary barriers and costs to practicing, without corresponding benefits of quality control. nobody practices law the way they take bar exams — answering questions from memory, as opposed to researching them.