Good to Evaluate Our Pet Notions

You may also like...

1 Response

  1. Jim Greiner says:

    Hi, Jeff, thanks so much for writing, and for sharing these ideas with us (as you had before).

    We agree that an organizational sociology/ethnography dimension to the sort of randomized evaluations we’re pursuing would be helpful. There are a couple of challenges to consider, but none undercut your basic point that what you suggest is highly worthwhile.

    The first is that we may change the behavior of the system if we observe it in a very, very obvious way; this is the classic Heisenberg uncertainty problem (with apologies to the physicists for abusing this phrase). Our sense is that it may be easier for an adjudicatory system to “forget” an ongoing numbers-based evaluation that it does not “see” day to day than it is for that system to “forget” evaluators in its buildings who watch its operations.

    Second, the kind of study you’re recommending feels as though it would be at least as hard to conduct as the randomized evaluation we’re pursuing. Herb Kritzer’s book Legal Advocacy: Lawyers and Nonlawyers at Work shows how hard one version of this is to do. We do not find many of the quantitative conclusions in Herb’s work persuasive for the reasons discussed in our paper, but I at least (I can’t speak for Cassandra here) found the observational reporting powerful. The historical research you’re also suggesting would seem similarly difficult.

    Which brings us to the third point: are Cassandra and I well-suited to do this kind of work? My guess is that the kind of organizational investigation, and the kind of observation Herb did in his book, is no simpler to do well than is a randomized evaluation. It definitely requires the cooperation of potentially reluctant or resistant governmental agencies. And it probably requires mastery of a certain, defined qualitative methodology; the fact that I don’t know whether such a methodology exists may itself say something about whether we are well-suited to pursue it. Cassandra and I can bring quant knowledge to the table, and I can add my background as a litigator. We would need to tread carefully if we undertook an exhaustive organizational sociology/ethnography if the type you suggest.

    Of course we think we can learn and do everything (who doesn’t?), but pursuing this might require an investment in learning investigative methodology as well as the time and effort to do any particular study. Is this an argument for additional co-authoring?

    Again, many thanks for writing!

    Jim & Cassandra