Team led by Sebastian Gombert wins one of two tracks at BEA 2024 shared task on predicting Item Difficulty and Item Response Time

Team led by Sebastian Gombert wins one of two tracks at BEA 2024 shared task on predicting Item Difficulty and Item Response Time

Artificial Intelligence, Assessment, Award, Computational Psychometrics, Conference, Higher Education, New Pub, Workshop
For standardized exams to be fair and reliable, they must include a diverse range of question difficulties to accurately assess test taker abilities. Additionally, it's crucial to balance the time allotted per question to avoid making the test unnecessarily rushed or sluggish. The goal of this year's BEA shared task (competition) was to build systems which could predict Item Difficulty and Item Response Time for items taken from the United States Medical Licensing Examination (USMLE). EduTec member Sebastian Gombert designed systems which are able to predict both variables simultaneously. These placed first out of 43 for predicting Item Difficulty and fitfth out of 34 for predicting Item Response Time. They use modified versions of established transformer language models in a multitask setup. A corresponding system description paper titled Predicting Item…
Read More
New Pub: From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study

New Pub: From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study

Artificial Intelligence, Assessment, Computational Psychometrics, Empirical Study, Feedback, Higher Education, Journal, Publication, Special Issue, Technical paper
How can we give students highly informative feedback on their essays using natural language processing? In our new paper, led by Sebastian Gombert, we present a case study on using GBERT and T5 models to generate feedback for educational psychology students. In this paper: ➡ We implemented a two-step pipeline that segments the essays and predicts codes from the segments. The codes are used to generate feedback texts informing the students about the correctness of their solutions and the content areas they need to improve. ➡ We used 689 manually labelled essays as training data for our models. We compared GBERT, T5, and bag-of-words baselines for both steps. The results showed that the transformer-based models outperformed the baselines in both steps. ➡ We evaluated the feedback with a learner cohort…
Read More
Workshop: Hyperchalk – How to implement Self-hosted Whiteboard Tasks @ JTEL Summer School 2023

Workshop: Hyperchalk – How to implement Self-hosted Whiteboard Tasks @ JTEL Summer School 2023

Artificial Intelligence, Computer-supported collaborative learning, Learning Analytics, Summer School, Workshop
In this workshop which Lukas Menzel and I gave at the seventeenth JTEL Summer School, we explored the possibilities of our self-implemented whiteboard tool Hyperchalk. Hyperchalk is a backend for Excalidraw which allows for integrating learning management systems via LTI and collecting a complete history of trace data. After a short kick-off presentation, we let the participants design their own learning activities using the whiteboard. All participants created little tasks that other participants then solved. As most participants had a strong background in teaching, these were inspired by practical experiences. The tasks involved various topics, from stochastics to K12-level geography. At the end of the workshop, we taught the participants how to administrate the tool and how to set it up on their own servers. Overall, it was a successful…
Read More
Workshop: Large Language Models for Feedback Generation @ JTEL Summer School 2023

Workshop: Large Language Models for Feedback Generation @ JTEL Summer School 2023

Artificial Intelligence, Feedback, Summer School, Workshop
At the seventeenth JTEL Summer School, Lukas Menzel and I had the pleasure of giving a workshop on the potentials and pitfalls of large language models for generating learner feedback. We kicked the event off with a general presentation on large language models. We explained the technical properties of well-known language models such as BERT or GPT. Following this, we went into different setups that can be used for feedback generation. On the one hand, this can involve training a BERT-based model to predict codes for input responses that trigger OnTask-style feedback rules. While this approach is stable regarding what feedback students receive, it is also inflexible, as such feedback cannot necessarily mirror all detailed intricacies that might occur in a student's response. For this reason, it can feel kind…
Read More
EC-TEL2022 best demo award goes to EduTec

EC-TEL2022 best demo award goes to EduTec

Award, Conference
The demo paper Superpowers in the Classroom: Hyperchalk is an Online Whiteboard for Learning Analytics Data Collection and the corresponding tool Hyperchalk by Lukas Menzel, Sebastian Gombert, Daniele Di Mitri and Hendrik Drachsler were awarded the best demo award at the European Conference on Technology Enhanced Learning 2022. Hyperchalk allows hosting collaborative whiteboards which can be used to collaborate in real-time. It collects rich user data which can be used to conduct learning analytics. Hyperchalk was initially developed for the ALICE project.
Read More
Sebastian Gombert @ EC-TEL 2022 Doctoral Consortium

Sebastian Gombert @ EC-TEL 2022 Doctoral Consortium

Artificial Intelligence, Conference, Event, Learning Analytics, Research topic
At EC-TEL 2022, Sebastian Gombert presented his doctoral project "Methods and perspectives for the automated analytic assessment of free-text responses in formative scenarios". The goal behind this project is to build free-text assessment systems which can code constructed responses analytically. While much of the past work in response scoring focused on predicting holistic scores, analytic scoring allows to code free-text responses for multiple different aspects. The goal behind these efforts is to provide learners and teachers with useful feedback on each of them. A more detailed project overview will be released as part of the EC-TEL 2022 doctoral consortium proceedings soon.
Read More
New Pub: Superpowers in the Classroom: Hyperchalk is an Online Whiteboard for Learning Analytics Data Collection

New Pub: Superpowers in the Classroom: Hyperchalk is an Online Whiteboard for Learning Analytics Data Collection

Computer-supported collaborative learning, Conference, Conference, Event, Further Education, General education, Higher Education, Learning Analytics, Lifelong Learning, Publication, School, Technical paper
A new system demonstration paper authored by Lukas Menzel, Sebastian Gombert, Daniele Di Mitri and Henrik Drachsler has been released as part of the ECTEL 2022 proceedings. In this paper, we present Hyperchalk, a self-hosted collaborative online whiteboard software. Similar to commercial solutions like Miro or Flinga, this software provides users with collaborative boards which they can use to draw, write or sketch together. However, unlike commercial solutions, Hyperchalk allows for collecting rich log data, which can be used to study the behaviour of its users and to allow Learning Analytics and studies on computer-supported collaborative learning. Moreover, Hyperchalk comes with a built-in replay mode which allows watching how users behave in its spaces. It supports the LTI1.3 standard, which enables seamless integration with learning management systems such as Moodle,…
Read More
Team of EduTec and TBA Members Wins Runners-Up Prize @ NAEP Reading Automated Scoring Challenge

Team of EduTec and TBA Members Wins Runners-Up Prize @ NAEP Reading Automated Scoring Challenge

Artificial Intelligence, Assessment, Award, Computational Psychometrics, School
The NAEP Reading Automated Scoring Challenge was a competition which was held in late 2021 by the National Center for Education (USA). The goal was to evaluate the applicability of natural language processing methodology to the task of scoring of large scale constructed response assessment data. In addition to this, participating models were also required to be interpretable and free of algorithmic bias against different student demographics. The data set which was used to evaluate the systems consisted of responses from 4th and 8th grade school student to 20 different reading comprehension tasks. More than a dozen different teams from universities and private assessment companies submitted contributions. Out of these, 12 contributions, including ours, were selected as eligible with respect to interpretability and algorithmic fairness. These contributions were then ranked…
Read More