Category: Publication

At BEA 2026, we organized the first shared task on rubric-based short answer scoring for German, introducing a new benchmark dataset spanning multiple STEM domains and designed to evaluate both in-domain performance and generalization to previously unseen questions. The goal was to better understand how modern NLP systems can interpret and apply textual scoring rubrics—an ability that closely mirrors how human assessors evaluate student responses. The shared task attracted multiple teams, who explored a wide range of approaches, including fine-tuned large language models, retrieval-augmented prompting, hybrid symbolic–neural systems, and ensemble methods. Across all four evaluation tracks, systems that explicitly incorporated rubric semantics consistently achieved the strongest performance, while scoring previously unseen questions remained the biggest challenge. We hope this benchmark provides a foundation for future research on more robust, interpretable,…

How can AI not only score student answers accurately, but also explain why it assigned a particular score? In our latest paper, we introduce AGRAA (Aspect-Grounded Rubric–Answer Alignment), a new framework for rubric-based assessment that represents rubric criteria as semantic subspaces. Instead of treating scoring as a standard classification problem, AGRAA measures how strongly a student's response aligns with the latent semantic aspects defined by each rubric level. We evaluate the approach on both short-answer and essay scoring benchmarks, where it consistently achieves performance highly competitive with strong transformer-based baselines. More importantly, the model naturally produces interpretable explanations by showing which rubric-defined aspects contributed to each scoring decision. We believe this work is a step toward automated assessment systems that are not only accurate, but also transparent, trustworthy, and aligned…

In a recent study, we explored how students’ conceptual understanding develops throughout a digitally supported chemistry unit on chemical kinetics. Working with approximately 300 upper-secondary students, we combined automated scoring of students’ written responses and other learning artifacts with network analysis techniques to model the growth of individual knowledge structures over time. The results showed that transformer-based language models can reliably identify and score chemistry-related knowledge elements from classroom data, enabling the construction of longitudinal knowledge networks that reflect how students connect scientific concepts. Several network characteristics were associated with students’ posttest performance, suggesting that these representations capture meaningful aspects of learning progress. The findings highlight the potential of combining automated assessment and learning analytics to provide teachers with real-time insights into students’ developing understanding and to support adaptive instructional…

At this year's JTEL Summer school, Sebastian Gombert and Amir Rajabi from DIPF hosted a workshop on Natural Language Processing (NLP) for Education and Research. Participants explored how modern language technologies can support both educational practice and scientific inquiry. The session introduced core NLP concepts and discussed applications ranging from automated assessment and feedback generation to classroom discourse analysis, literature synthesis, and retrieval-augmented generation. Following the introductory presentation, participants worked collaboratively in small groups to design NLP-based solutions for real-world challenges. Drawing on both provided examples and their own research interests, attendees developed concepts for systems such as educational recommender tools, feedback assistants, and research support applications. Interactive collaboration was supported through discussion, sketching, and digital whiteboarding. The workshop highlighted not only the growing potential of large language models and…

On 01. Mai, Longwei Cong presented his paper “Automatic Short Answer Grading with LLMs: From Memorization to Reasoning” at the 16th International Conference on Learning Analytics and Knowledge. The paper examines the performance of fine-tuned PLMs and LLMs across different dataset sizes and compares them with prompt-based approaches for automatic short answer grading. The results show that fine-tuned LLMs and rubric-based prompting can match or even exceed the performance of BERT-based models. In particular, rubric-based prompting with open-weight models can deliver competitive results without requiring annotated training data or hardware-intensive fine-tuning, while also helping to address data protection concerns. This work provides empirical evidence for the role of LLMs in automatic short answer grading and opens up future research directions on resource-efficient, interpretable, and reasoning-driven grading. You can find the…

Rubrics are everywhere in education — but surprisingly, most AI systems for grading short answers barely use them. In our recent paper, Are Rubrics All You Need?, presented by Sebastian Gombert at the LAK 2026, we introduce rubric-based automatic short-answer scoring, a new approach where AI models explicitly align student answers with rubric criteria instead of treating grading as a black-box classification problem. We propose two novel architectures, GRAASP and ToLeGRAA, which use transformer-based alignment mechanisms to compare learner responses directly against rubric descriptions. Across German and English benchmark datasets, the models achieved highly competitive performance and transferred better to unseen questions than traditional instance-based classifiers. Particularly exciting is ToLeGRAA’s ability to generate token-level alignment maps, making it possible to visualize which parts of a student answer correspond to specific…

Are you a researcher in AI in education and/or natural language processing? Do you like shared tasks and machine learning competitions like the yearly SemEval tasks or Kaggle competitions? Then you might consider participating in the BEA 2026 Shared Task on Rubric-based Short Answer Scoring for German, co-organized by us and colleagues from IPN - Leibniz-Institut für die Pädagogik der Naturwissenschaften und Mathematik. For this shared task, we look at the natural language understanding task of rubric-based short answer scoring, a task for which models must score short answers to open-ended assessment questions with the help of provided textual scoring rubrics. The dataset we provide for this was collected in authentic German school contexts and scored by domain experts as part of the ALICE project funded by the Leibniz-Gemeinschaft. As…