[New Pub] Exploratory Bayesian Multidimensional IRT with Rater Effects for Scoring Generative AI Usage
Artificial Intelligence, Assessment, Computational Psychometrics, Conference, Critical Online Reasoning, Higher Education, Publication, Research Methods
At AIME-Con 2026 (October 5–7, Pittsburgh), Leon Camus presented the paper "Exploratory Bayesian Multidimensional IRT with Rater Effects for Scoring Generative AI Usage". Scoring frameworks usually treat students' AI use as a single scale from little to a lot. This hides the difference between a student who uses a chatbot as a research assistant and one who hands the whole task to it. The paper introduces a multidimensional Bayesian IRT model with rater effects that captures these different usage patterns without predefining which rubric items belong to which dimension. Applied to about 18,000 ratings of 251 German undergraduates working on Critical Online Reasoning tasks, the model identifies three dimensions of AI use: active prompting effort, delegation of substantive content, and AI-delivered citation use. Students working in their own academic domain prompted more,…
