The study is described in a Draft document for 2024 published on the SSRN website, formerly known as the Social Science Research Network. It is currently under revisions with the academic journal Management Science.
The researchers found an end-of-the-alphabet rating bias across a wide range of subjects. However, the assessment penalty is more pronounced in the social sciences and humanities than in engineering, science, and medicine.
In addition to lower grades, the researchers also found that students at the bottom of the alphabet received more negative and rude comments. For example, “why are there no answers to questions 2 and 3? You’re setting yourself up for a failing grade’ and ‘NEVER DO THIS AGAIN’. Top students are more likely to get: “Much better work on that draft (Student Name)! Thanks!”
The researchers can’t prove exactly why extra points are deducted for Wilson of the World, but they suspect it’s because the instructors—mostly graduate students at the unnamed university in this study—have heavy grading workloads and get tired and irritable, especially after as they evaluate the 50th student in a row. Even before the e-grading era, it’s very likely that instructors weren’t as fair to the students at the bottom of the paper pile. But in the paper world, the student’s position in the pile was always changing depending on when the papers were handed in and how the instructors took them. No student was likely to be at the bottom of the pile every time. In the LMS world, the letters U, V, W, X, Y, and Z almost always are.
Another theory mentioned by the authors in the paper is that instructors may feel the need to be stricter if they have already given a string of A’s, so as not to be too generous with high grades. Students at the bottom of the alphabet may be victims of well-intentioned efforts to curb grade inflation. It is also possible that instructors are too generous with students at the top of the alphabet, but grade more accurately as they go along. Either way, students at the bottom are graded differently.
Some college professors seem to be aware of their human frailty. In 2018, one posted on a message board in Canvas, asking the company to randomize the gradebook. “For me, biases start to creep in with fatigue,” the instructor wrote. “I rate a few, move away from it, rate a few more, give it a break. Or that’s the goal when I’m not facing a deadline.”
If you’ve read this far, you might be wondering how the researchers knew that the U to Z student grades were unfair. Maybe they are comparatively worse students? But the researchers matched Canvas grades with student records in the registrar’s office and were able to control for a host of student characteristics, from high school grades and college GPA to race, ethnicity, gender, family background and income. Last names at the end of the alphabet consistently received lower grades even among similar students who were graded by the same instructor.
The researchers also found that a minority of instructors tinkered with the default settings and graded in reverse alphabetical order, from Z to A. This produced the exact opposite results; students with names at the end of the alphabet earned higher grades, while grades for A, B, and C surnames were lower.
A penchant for last names at the end of the alphabet is probably not unique to students using the Canvas LMS. All four major LMS companies, which collectively control 90 percent of the U.S. and Canadian market of more than 48 million students, rank submissions alphabetically for grading, according to the researchers. Even Coursera, a stand-alone online learning platform, does it this way.
Wang’s solution is to shake things up and have the LMS present students’ work for grading in random order. Indeed, Canvas added a randomization option for instructors in May 2024after the company saw a draft of that University of Michigan study. “This was something we had on our radar and had heard from some users, but we haven’t finalized it yet,” a company spokesperson said. “The report from the University of Michigan definitely made this work a top priority.”
However, the default remains alphabetical and instructors must navigate to the settings to change it. (Changing this default, according to the study’s authors, has “low visibility” in the site’s system settings.) I hope this story helps get the word out.
This story about learning management systems was written by Jill Barshey and produced by The Hechinger Reportan independent, non-profit news organization focused on inequality and innovation in education. Sign up for Evidence points and others Hechinger Bulletins.
