Knowing your ABCs is essential to academic success, but having a last name starting with A, B or C might also help make the grade.
An analysis by University of Michigan researchers of more than 30 million grading records from U-M finds students with alphabetically lower-ranked names receive lower grades. This is due to sequential grading biases and the default order of students’ submissions in Canvas—the most widely used online learning management system—which is based on the alphabetical rank of their surnames.
What’s more, they find, those alphabetically disadvantaged students receive comments that are notably more negative and less polite, and exhibit lower grading quality measured by post-grade complaints from students.
“We spend a lot of time thinking about how to make the grading fair and accurate, but even for me, it was really surprising,” said Jun Li, associate professor of technology and operations at U-M’s Ross School of Business who co-authored the study with doctoral students Jiaxin Pei from U-M’s School of Information and Helen (Zhihan) Wang from Ross.
“It didn’t occur to us until we looked at the data and realized that sequence makes a difference.”
The researchers collected available historical data of all programs, students and assignments on Canvas from the fall 2014 semester to the summer 2022 semester. They supplemented the Canvas data with university registrar data, which contains detailed information about students’ backgrounds, demographics, and learning trajectories at the university.
Although the data is from U-M, the researchers say the findings can be generalized across institutions and courses. They are driven by a common design issue of learning management systems—the default setting of ranking students’ assignments alphabetically by their names.
Their research uncovered a clear pattern of a decline in grading quality as graders evaluate more assignments. According to Wang, students whose surnames start with A, B, C, D or E received a 0.3-point higher grade out of 100 possible points than compared to when they were graded randomly. Likewise, students with later-in-the-alphabet surnames received a 0.3-point lower grade—creating a 0.6-point gap.
Wang notes for a small group of graders (about 5%) that grade from Z to A, the grade gap flips as expected: A-E students are worse off, while W-Z students receive higher grades relative to what they would receive when graded randomly. Such observations confirm their hypothesis that it’s the order of grading that leads to the initial gap in grades.
A 0.6-point difference might seem small, but such a disparity did affect students’ course grade-point averages, which negatively influences opportunities in their respective career paths.
“Our conclusion is this may be something that happened unconsciously by the graders that’s actually creating a real social impact,” Wang said.
Pei says the idea for the study came up during a discussion he had with Wang in which they were talking about their research: She studies educational technology and he studies artificial intelligence. He observed that a fundamental task of machine learning is data labeling, also a sequential task that can be long and tedious—but one that is randomized.
It got them thinking about educational systems like Canvas and led to some pilot studies to see if there was any disparity among grades based on the amount of time spent in the task of grading.
“We kind of suspect that fatigue is one of the major factors that is driving this effect, because when you’re working on something for a long period of time, you get tired and then you start to lose your attention and your cognitive abilities are dropping,” Pei said.
The researchers note the option exists to grade the assignments in a random order, and some educators do, but alphabetical order is the default mode in Canvas and other online learning management systems. One simple fix would be to make random order the default setting.
They also suggest academic institutions could hire more graders for larger classes, distribute the workload among more people or train them to be aware of and lessen the bias while grading.
Li, Wang and Pei have been sharing their research at conferences and it’s been positively received—many are impressed by their work though it confirms suspicions many harbor. One reaction in particular stands out to Li—no doubt an information-age wrinkle on “the dog ate my homework” excuse.
“A college student emailed us afterward asking us to share the paper with him,” she said. “He mentioned that his last name started with W. He’s going to tell his parents it’s not because of him—it’s because of his last name.”
The study is under review by the journal Management Science and currently available as a working paper.
More information:
Zhihan (Helen) Wang et al, 30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity (2023). On SSRN: ssrn.com/abstract=4603146
Provided by
University of Michigan
Citation:
Researchers find lower grades given to students with surnames that come later in alphabetical order (2024, April 17)
