At AIME-Con 2026 (October 5–7, Pittsburgh), Leon Camus presented the paper “Exploratory Bayesian Multidimensional IRT with Rater Effects for Scoring Generative AI Usage”.
Scoring frameworks usually treat students’ AI use as a single scale from little to a lot. This hides the difference between a student who uses a chatbot as a research assistant and one who hands the whole task to it. The paper introduces a multidimensional Bayesian IRT model with rater effects that captures these different usage patterns without predefining which rubric items belong to which dimension.
Applied to about 18,000 ratings of 251 German undergraduates working on Critical Online Reasoning tasks, the model identifies three dimensions of AI use: active prompting effort, delegation of substantive content, and AI-delivered citation use. Students working in their own academic domain prompted more, sought more citations, and delegated less, a “research assistant” pattern of AI use.
The paper is available via the following link: Exploratory Bayesian Multidimensional IRT with Rater Effects for Scoring Generative AI Usage – ACL Anthology