The AI models addressed female students more gently and used more first-person pronouns. (“I like your confidence in voicing your opinion!”) Students identified as unmotivated were met with upbeat encouragement. Conversely, students described as high achievers or motivated are more likely to receive direct, critical suggestions aimed at improving their work.
Different words for different students
In other words, the AI feedback was different both in tone and in terms of the expectations it had for the student. The paper, “Marked pedagogies: Exploring language biases in personalized feedback in automated writing”, has not yet been published in a peer-reviewed journal, but was nominated for Best Paper at 16th International Conference on Learning and Knowledge Analytics in Norway, where it is scheduled to be presented on April 30.
The researchers described the feedback results as showing a “positive feedback bias” and a “feedback bias” – offering more praise and less criticism to some groups of students. Although the differences in each individual piece of written feedback may be hard to spot, patterns were evident across hundreds of essays.
The researchers believe that the artificial intelligence changes its feedback on identical essays because the models are trained on a huge amount of human language. Human teachers may also tone down criticism when responding to students from certain backgrounds, sometimes because they don’t want to appear unfair or discouraging. “They capture the biases that people exhibit,” said Mei Tan, the study’s lead author and a doctoral student at the Stanford Graduate School of Education.
At first glance, feedback differences may not seem harmful. More encouragement can boost a student’s confidence. Many educators argue that culturally responsive teaching—acknowledging students’ identities and experiences—can increase student engagement in school.
But there is a trade-off.
If some students are constantly shielded from criticism while others are forced to sharpen their arguments, the result can be unequal opportunities for improvement. Praise can be motivating, but it’s no substitute for the kind of specific, direct feedback that helps students grow as writers. Tanya Baker, executive director of the National Writing Project, a nonprofit organization, recently heard a presentation of this study and said she worries that black and Latino students may not be “forced to learn” to write better.
This raises a difficult question for schools as they adopt AI tools: When does useful personalization cross the line into harmful stereotyping?
Of course, it’s unlikely that teachers would explicitly tell AI systems a student’s race or background the way the researchers did in this experiment. But that doesn’t solve the problem, the Stanford researchers said. Many educational databases and learning platforms already collect detailed information about students, from previous achievements to language status. As AI builds into these systems, it can access far more context than a teacher would knowingly provide. And even without explicit tags, AI can sometimes infer aspects of identity from the writing itself.
The bigger problem is that AI systems are not neutral educators. Even the regular feedback response—where the researchers did not describe the student’s personal characteristics—used a special approach to written instruction. Tan described it as quite discouraging and focused on making adjustments. “Perhaps the bottom line is that we shouldn’t leave pedagogy to the big language model,” Tan said. “People should be in control.”
Tan recommends that teachers review written feedback before forwarding it to students. But one of the benefits of AI feedback is that it’s instantaneous. If the teacher has to review it first, it slows it down and potentially undermines its effectiveness.
AI also offers the potential for personalization. The risk is that, without careful consideration, this personalization can lower the bar for some students while raising it for others.
This story about AI bias is produced by The Hechinger Reportan independent, nonprofit news organization that covers education. Sign up for Evidence points and others Hechinger Bulletins.
