Examining Law School Exams

pen6a.jpgThere are a lot of really good discussions going on in the blogosphere about law school exams recently.

Ann Althouse asks whether exams are a rewarding educational experience in and of themselves for students. Jonathan Adler offers his thoughts here. Rick Garnett chimes in at PrawfsBlawg.

In most law school courses, the grade is based on one final exam given at the end of the semester. Eugene Volokh offers a defense of this practice:

Naturally, some students might value the greater accuracy of the multi-exam approach, and the lower risk to them that this approach provides (since they don’t need to worry as much about tanking a class because they screw up one exam). But for many students the downside of yet more studying and yet more worrying exceeds the upside of less risk and better measurement accuracy. And on top of that, grades are on balance mostly a zero-sum game — more accuracy of testing will just rearrange the As, Bs, and Cs, so as many students will lose as will win. Increased time needed to study for exams is a net loss for students as a whole. Therefore, you’d expect that more accurate measurement devices will please only a few students, while greater need to study will displease many more students.

So professors don’t want to institute more exams, even if having more exams provides more accurate measurement. More students, I suspect, don’t want to institute more exams, even if having more exams provides more accurate measurement. Who would predictably benefit from more accurate measurement, and lose nothing from it? Employers, and perhaps indirectly the legal system generally.

My take on exams is that the big problem with them is the format, not the one-exam-decides-it-all system. Exams are given under immense time pressure, thus favoring students who are able to analyze and write cogently in a very quick timeframe. This is a good skill, although it is significantly undervalued in practice, where law firms reward associates based on billable hours. Ironically, law firms therefore should select students who don’t do well on law school exams, since it will take these students longer to complete assignments, and hence generate more billable hours. In the land of the billable hour, the inefficient lawyer is king.

Perhaps the most appropriate thing for law school exams to measure is a student’s ability to produce a thoughtful legal memo under less onerous time constraints. For some classes, I experimented with take home exams where students had 24 hours to complete them. If 24 hours is enough time for Keifer Sutherland to save the world, it’s enough time for a student to write a short memo. I figured that this would give students more opportunity to think and analyze rather than just write in a frenzy. However, there are some problems with take home exams. First, take home exams increase the possibility of cheating. Second, many students find them to be especially onerous. Although my take home exams were designed to be completed in about four to six hours of work time (spread throughout the day so students had some time away from the exam), students would often spend the full 24 hours toiling on the exam. Many weary-eyed students told me that they preferred a three-hours-and-you’re-done exam rather than pulling an all-nighter on my 24 hour take home exam. Finally, I was surprised to find that the quality of the take home exams wasn’t that much better than the in class exams. I can’t explain why this was the case; it cut against everything I had theorized.

Are there better ways to measure student performance in a class? In thinking about this question, Volokh’s point about efficiency is worth thinking seriously about. More accurate metrics of performance often have costs in time and stress, and sometimes the less perfect but more efficient metric is best. Part of the analysis depends on the answer to Ann Althouse’s question. If exams are in and of themselves a rewarding educational experience (rather than just a metric of academic performance), then this is a factor that must be weighed in the balance. I remain uneasy about traditional in class exams, but I have not yet hit on an alternative that is significantly better. Take home exams might be a marginal improvement, and in the future I might experiment more with take home exams.

You may also like...

12 Responses

  1. Did you put a length limit on the 24-hour exams? As a student, I found 24-hour exams with strict (both in magnitude and in enforcement) word or page limits to be the most humane examination system, followed by scheduled closed-book exams, then by scheduled open-book exams. Take-home exams without length limits (or limits that weren’t very constraining) were dead last. Length limits diverted more time into thinking and less into writing, put an upper bound on the number of topics worth exploring, and forced hard (but thought-provoking) choices about which of several points were worth making.

  2. Yes, I put a rather strict length limit on the exam — I think it was something like 15 pages, double spaced, courier font. I’ve often found that longer exams aren’t generally better, just wordier. My length restrictions were difficult, and they probably required students to do a bit of editing and trimming, which in my mind is a very good skill to learn.

    I also established a penalty system for late exams. My policy: “If an exam is turned in late (beyond the 24 hours allotted), I will deduct a third of a grade. Additionally, I will continue deducting a third of a grade every 2 hours until the exam is turned in.” Surprisingly, even under the penalty system, about 10% of the exams came in late. One even came in over 6 hours late!

  3. MR says:

    I agree with most of this, but I must take issue with the notion that firms value billable hours to the exclusion of quick analytical thinking.

    First, such thinking makes for better litigators, which means you can send associates to certain un-fun tasks with more confidence. Second, there is value in handling many matters at once, which can be done only with efficiency. Third, a lot of work is last minute, and if the work isn’t getting done, billable hours are useless. Fourth, the modern trend is that clients want efficiency and are actually looking at bills these days.

    I personally preferred the three-hour exam for the most part. Quick and over quickly. I tended to fill the time alotted working, so 24 hours = 24 hours of work.

  4. MR — Everything you say I agree with — analytical thinking and getting work done efficiently are some of the most valuable skills a lawyer can have. But with the structure of many law firms these days, billables seem to be such a large part of what is expected of associates.

    There are some people who can work very intensely and well in short bursts of time. They are very valuable for litigation, but after a very productive burst of work, their energy is spent. Few people can work like this for 60+ hours per week. Their billables will look terrible, and they’ll probably start getting pressure from the firm to up their billables.

    Of course, firms may say that they value great efficiency, but what matters are the incentives law firms create. And at least from what I’ve seen and read, the incentives are skewed toward the billable hour.

  5. 1L says:

    I think what Mr. Volokh’s comments suggest is that students are being done a favor by having one exam. As a 1L(so my experience is limited) I find this to be untrue. Doesn’t this approach really just make it easier for the professors when it comes to grading. Having a more accurate measure should trump any supposed increased pressure on the students. Law School is all pressure, all the time. Whether you are studying for an exam or preparing for class the pressure still exists. What should be considered is producing the best lawyers and doing the best job to accurately determine who is the most knowledgeable.

    Giving the wrong grade to the wrong person because a student had a bad day or the subjective nature of a pressure packed 4.5 hour exam to determine 14 weeks of work when a better alternative exists seems to be what professors should be striving for and explaining why they MUST test multiple times despite the additional pressures because it is the most accurate way to prove who is the correct student that is exceeding in the class.

    Everyone knows how much weight grades can have in a person getting a job and determining the lives and futures of the students. How could relieving stress be a proper suggestion, when a person’s future could be at stake(I understand I am being dramatic),by people who have been through the system, usually at the most prestigious schools in the country?

    To suggest that it is a zero sum game and you will only be rearranging the As, Bs and Cs is like suggesting that we shouldn’t have the 4th Amendment because it really only protects criminals.

    I find this all a little convenient and as a student it seems that the real ease of burden is on Professors, many who are teaching from the same book and notes that they have been using for years.

    I guess you can tell I just finished my spring exams today.

  6. Howard Wasserman says:

    I am a big believer in what you do–24 hours, at-home, page-limits. Not sure what to do about cheating, of course. But I like the product that results and I like taking away the time crunch. I end up spending a lot of time during the semester repeating that they should not/cannot spend 24 hours on it, that it is not that daunting so don’t freak out, etc.–sometimes students even believe me.

    I also have been experimenting in smallish upper-level classes with a model proposed by one of my colleagues. We might call it a paper/exam combo–a semester-long project, written as a paper, but answering one of several specific questions that require students to engage in the deep structure and application of the doctrine.

  7. I misread “I will continue deducting a third of a grade every 2 hours until the exam is turned in,” as saying, “I will continue deducting a third of the total grade every 2 hours until the exam is turned in,” and wondered to myself, “Did you give the 6-hour-late exam a negative grade?”

  8. Amy says:

    I can see the necessity of page limits on take-homes, but for those not naturally concise, they actually created more work – I would literally spend half the allotted time editing.

    What about 8-hour take-homes – i.e., basically a normal-length exam with twice as much time to write?

  9. Bruce says:

    Dan, I’m with MR — I did not perceive any pressure when I was at a law firm to work more slowly. Quite the opposite. I think the reason for this is related to the reason, which puzzled me at first, why cab drivers in NYC who get paid by the mile drive so fast. The reason is that, the quicker they get you to your destination, the sooner they can pick up another fare. The number of fares per hour, not the distance they travel, is the primary limit on their earnings.

    Same with law firms. The number of repeat clients per year, not the hours you bill to them, is typically the limiting factor on law firm profits. Clients usually want an answer (or a motion, or what have you) as fast and as cheaply as possible. An associate who does a task in 1 hour as opposed to 2 makes the client happy on both counts. Happy clients give you more business. Also that associate is available sooner to make other clients happy. And if the junior associates are good and efficient, you can delegate more work down to them, which increases the cap on difficult work that would otherwise be imposed by the limits on partner availability. Inefficient associates, on the other hand, mean you have to hire more of them to do the same number of tasks in the same amount of time, and that work is likely to be of lower quality. Hiring more associates is a huge expense. This is why associate salaries keep going up; because of benefits and overhead, it’s cheaper to pay $20,000 more per year and ask for insane hours from each associate (often way more than 60 per week — which I assure you is possible), even if attrition rates are high, than hire more associates at lower pay and have them work fewer hours.

  10. Anjesh says:

    Hi,can i have a model of the test papers for november 2006 exams

  11. Monica says:

    I’m a student, aspiring prof, and three-hour law exams are threatening to ruin my career.

    The purpose of a law exam should not be to assess who will the quickest litigator for a big firm. Many law students, amazingly enough, do not aspire to be litigators at big firms. At my law school, one does not need good grades to be a litigator at a big firm, but you need good grades to do things like teaching and impact litigation. Neither of those call on the “skills” 3-4-hour law exams do.

    I am in favor of the 24-hr page-limited exam. It actually tests a skill necessary for lawyering: concision.

  12. Spurious_George says:

    All it would take for a law exam to be rewarding is for the best grades to go to the students who worked the hardest through consistent and effective class preparation and exam preparation, who understood the material the best (including the most difficult concepts studied), and who managed their time well enough that they did not have to cram and did not experience any easily preventable personal problems that would have hindered their exam performance. But this is not what happens, and that is why law school exams are not satisfying. If exams were satisfying then, you would almost certainly get a good grade if the following were true:

    1) You were consistently well-prepared for class, and you were a regular participant, and you gave good answers to difficult questions even when you weren’t specifically on call;

    2) You finished your outline as soon as your last class was finished, and you spent a few days revising your outline, meeting with the professor to clarify any issues that you didn’t fully understand (which should only be a small number of issues and probably on the most difficult issues, since you kept up with the work from start to finish and you were not cramming);

    3) You spent some time thinking about how the professor would reasonably test you on what you learned over the past 15 weeks and formulated some general test-taking strategies;

    4) You got a good nights sleep before the exam and generally took care of yourself (e.g., good eating habits, no hangovers, no breaking up with your s/o the night before the exam, no personal problems cropping that were due to your own irresponsibility, and you were on top of your other classes so you had time to focus on this exam); and

    5) The vast majority of your classmates couldn’t say their performance met these standards.

    My first year grades were disappointing. I spent a lot of time thinking about what I could do better this year (my 2L year). The above is the plan I came up with. Factor #5 is obviously out of my control since I can’t choose whether my classmates will be smart, diligent and responsible. But based on what I know of my classmates’ personalities and personal lives; and the general level of class participation, and the number of students who skip weeks of classes; and the number of students who either pass when called on or can’t give an insightful answer when called on; and how those same students only become regular attendees/ participants during the last two weeks of class — based on these factors, I think it is safe to say that I was way ahead of the curve w/r/t criterion #5 in all of my classes this semester. In terms of preparation, my best class was Insurance Law. At the risk of sounding arrogant, my performance based on these criteria was unimpeachable. And my classmates’ performance in the aggregate was shameful. There was no attendance requirement. The professor became seriously ill in the middle of the semester — he died before the end of the semester — and had to be replaced by an adjunct who had a full-time firm job. The class was rescheduled from 10:00 a.m. to 7:50 a.m. to accommodate her schedule. As a result, about half the class failed to show up for most of the remaining classes. Very few students were regular and willing participants. I did not miss one class. I outlined almost as soon as each class had ended and had plenty of time to revise and review my outline before the final. I just got my grade, and it was a low B. The exam was severely time-pressured, and it did not test the most difficult subjects. I am assuming that the professor watered down the issues because she felt sorry for the students since the situation was so unfortunate. But I did not get a chance to distinguish myself from other students. Also, the test didn’t really test us on what I had learned. What I learned was what the most prevalent issues in insurance law are and how appellate courts analyze those issues. And I was ready to spot issues and analyze them using the same tools that appellate courts use. And I could discuss these concepts either very generally, or in case-specific detail, so that I could analyze either a somewhat unfamiliar issue, or a very familiar issue. None of this did me any good. A low B at my school is a below-average grade, especially in a small class where the curve is not as rigid as in larger classes. This outcome makes no sense to me. I learned what I was encouraged to learn, so I should have been tested on what I learned. I have not received my other grades yet, and I am dreading it because Insurance was my “best” subject.