How can we give students highly informative feedback on their essays using natural language processing?
In our new paper, led by Sebastian Gombert, we present a case study on using GBERT and T5 models to generate feedback for educational psychology students.
In this paper:
➡ We implemented a two-step pipeline that segments the essays and predicts codes from the segments. The codes are used to generate feedback texts informing the students about the correctness of their solutions and the content areas they need to improve.
➡ We used 689 manually labelled essays as training data for our models. We compared GBERT, T5, and bag-of-words baselines for both steps. The results showed that the transformer-based models outperformed the baselines in both steps.
➡ We evaluated the feedback with a learner cohort at Goethe University using a randomised controlled trial. The control group received essential feedback, while the treatment group received highly informative feedback based on our pipeline. We used a six-item survey to measure the perception of feedback.
➡ We found that highly informative feedback had positive effects in terms of helpfulness and reflection. The students in the treatment group reported higher satisfaction, usefulness, and learning levels than those in the control group.
➡ Our paper demonstrates the potential of natural language processing for providing highly informative feedback on student essays. We hope our work will inspire more research and practice in this area.
You can read the full paper here: https://link.springer.com/article/10.1007/s40593-023-00387-6