Category: New Pub

How much does the wording of a prompt matter when AI scores students’ answers? In our paper, we investigated this using three open-weight language models and two English-language short-answer scoring benchmarks. We compared human-written prompts with prompts automatically refined using ProTeGi, which analyses scoring errors and proposes improved instructions. Across 135 paired comparisons, optimisation delivered a median improvement of around two F1 points. Beyond improving performance, it also revealed patterns in effective prompt design: optimised prompts used more words that make logical relationships explicit, such as “if” and “because,” and shorter noun phrases. These features were also associated with better scoring performance, while more directive verbs were associated with poorer results. Our findings suggest that clear logical structure and simpler phrasing are promising principles for prompt design, and that automatic…

At BEA 2026, we organized the first shared task on rubric-based short answer scoring for German, introducing a new benchmark dataset spanning multiple STEM domains and designed to evaluate both in-domain performance and generalization to previously unseen questions. The goal was to better understand how modern NLP systems can interpret and apply textual scoring rubrics—an ability that closely mirrors how human assessors evaluate student responses. The shared task attracted multiple teams, who explored a wide range of approaches, including fine-tuned large language models, retrieval-augmented prompting, hybrid symbolic–neural systems, and ensemble methods. Across all four evaluation tracks, systems that explicitly incorporated rubric semantics consistently achieved the strongest performance, while scoring previously unseen questions remained the biggest challenge. We hope this benchmark provides a foundation for future research on more robust, interpretable,…

How can AI not only score student answers accurately, but also explain why it assigned a particular score? In our latest paper, we introduce AGRAA (Aspect-Grounded Rubric–Answer Alignment), a new framework for rubric-based assessment that represents rubric criteria as semantic subspaces. Instead of treating scoring as a standard classification problem, AGRAA measures how strongly a student's response aligns with the latent semantic aspects defined by each rubric level. We evaluate the approach on both short-answer and essay scoring benchmarks, where it consistently achieves performance highly competitive with strong transformer-based baselines. More importantly, the model naturally produces interpretable explanations by showing which rubric-defined aspects contributed to each scoring decision. We believe this work is a step toward automated assessment systems that are not only accurate, but also transparent, trustworthy, and aligned…

In a recent study, we explored how students’ conceptual understanding develops throughout a digitally supported chemistry unit on chemical kinetics. Working with approximately 300 upper-secondary students, we combined automated scoring of students’ written responses and other learning artifacts with network analysis techniques to model the growth of individual knowledge structures over time. The results showed that transformer-based language models can reliably identify and score chemistry-related knowledge elements from classroom data, enabling the construction of longitudinal knowledge networks that reflect how students connect scientific concepts. Several network characteristics were associated with students’ posttest performance, suggesting that these representations capture meaningful aspects of learning progress. The findings highlight the potential of combining automated assessment and learning analytics to provide teachers with real-time insights into students’ developing understanding and to support adaptive instructional…

At this year's JTEL Summer school, Sebastian Gombert and Amir Rajabi from DIPF hosted a workshop on Natural Language Processing (NLP) for Education and Research. Participants explored how modern language technologies can support both educational practice and scientific inquiry. The session introduced core NLP concepts and discussed applications ranging from automated assessment and feedback generation to classroom discourse analysis, literature synthesis, and retrieval-augmented generation. Following the introductory presentation, participants worked collaboratively in small groups to design NLP-based solutions for real-world challenges. Drawing on both provided examples and their own research interests, attendees developed concepts for systems such as educational recommender tools, feedback assistants, and research support applications. Interactive collaboration was supported through discussion, sketching, and digital whiteboarding. The workshop highlighted not only the growing potential of large language models and…

On 01. Mai, Longwei Cong presented his paper “Automatic Short Answer Grading with LLMs: From Memorization to Reasoning” at the 16th International Conference on Learning Analytics and Knowledge. The paper examines the performance of fine-tuned PLMs and LLMs across different dataset sizes and compares them with prompt-based approaches for automatic short answer grading. The results show that fine-tuned LLMs and rubric-based prompting can match or even exceed the performance of BERT-based models. In particular, rubric-based prompting with open-weight models can deliver competitive results without requiring annotated training data or hardware-intensive fine-tuning, while also helping to address data protection concerns. This work provides empirical evidence for the role of LLMs in automatic short answer grading and opens up future research directions on resource-efficient, interpretable, and reasoning-driven grading. You can find the…