At the seventeenth JTEL Summer School, Lukas Menzel and I had the pleasure of giving a workshop on the potentials and pitfalls of large language models for generating learner feedback. We kicked the event off with a general presentation on large language models. We explained the technical properties of well-known language models such as BERT or GPT. Following this, we went into different setups that can be used for feedback generation. On the one hand, this can involve training a BERT-based model to predict codes for input responses that trigger OnTask-style feedback rules. While this approach is stable regarding what feedback students receive, it is also inflexible, as such feedback cannot necessarily mirror all detailed intricacies that might occur in a student’s response. For this reason, it can feel kind of robotic. To contrast this, we then went into feedback generated through GPT-like models. This feedback feels more human as it is more flexible in terms of addressing the content of a student response. However, given that GPT-like models are stochastic parrots, such feedback is not guaranteed to be factually correct. We discussed possible solutions that could help to mitigate these problems, such as using GPT as a summarizer that compiles predictions from other components such as a student model or external knowledge bases into coherent feedback texts. At the end of the workshop, we had a vivid discussion on the chances and problems that might arise with such feedback.
