Grading Lessons from Cognitive Psychology
We’re (hopefully) nearing the end of law school grading season. Personally, I take the Macbeth approach: “if it were done when ’tis done, then ’twere well it were done quickly.” In part, this is because I find grading unpleasant. I’m nervous about being unfair and inconsistent (and I also don’t want to get trolled by my students for being late).
There’s no avoiding that the grades we give make a substantial difference in our students’ near-term career prospects. While this adds to the stress to “get it right,” there is relatively little discussion in legal academia about how we grade. And although there are many different ways to grade, cognitive science provides at least two suggestions that seem broadly applicable.
First, grade by question, not by exam.
In his recent book, Thinking, Fast and Slow, Daniel Kahneman discusses grading. He describes how early in his career he would grade exams in the “conventional” way, “pick[ing] up one test booklet at a time and read[ing] all that student’s essays in immediate succession, grading them as I went.”
The problem with grading by exam is that it leaves the professor at the mercy of the “halo effect,” where the “first question . . . scored had a disproportionate effect on the overall grade.” Since Kahneman won a Noble Prize for his behavioral economics research while I once read a book about it, I’ll just quote him a bit more:
“The mechanism was simple: if I had given a high score to the first essay, I gave the student the benefit of the doubt whenever I encountered a vague or ambiguous statement later on . . . if a student had written two essays, one strong and one weak, I would end up with different final grades depending on which essay I read first.”
Khaneman’s solution: grade by question, not by exam.
Kahneman goes on to note that even knowing how well a student did on earlier questions on that same exam (for instance by writing the points earned on the front of the exam) can influence the grader, and therefore it’s best to put the point score somewhere not readily visible, like on the inside page. This all dovetails with why we grade exams blind: we don’t want to be influenced by our preconceived notions of student performance. Similarly, we should grade each question “blind,” uninfluenced by the students past performance on the exam itself.
Second, randomize the grading order across questions.
While grading by question eliminates the halo effect, it doesn’t eliminate another cognitive bias: the desire for regular distributions. For instance, if you are scoring a question out of five points, and you’ve given out fives to the past three exams, you’re more likely to give the fourth exam a lower score, regardless of how good the answer is. (Full disclosure: the author of the prior link, Jacoba Urist, is my sister).
Robert Shiller (who taught me behavioral economics), provides the solution: randomize exam order across question. That means that once you’ve graded all of the question ones, shuffle the papers and reorder the exams to grade the question twos.
These techniques won’t necessarily make grading any less nerve-wracking (or more fun), but they might make it a little more fair.
Anyone else have further grading tips?
Photo Credit: Wikipedia.