Sebastian Gombert, Author at EduTec Science

Jul12024 by Sebastian GombertNo Comments

New pub: Predicting Item Difficulty and Item Response Time with Scalar-mixed Transformer Encoder Models and Rational Network Regression Heads

Artificial Intelligence, Assessment, Computational Psychometrics, Conference, Higher Education, Publication, Workshop

In a contribution to the BEA 2024 Shared Task, we addressed the challenge of predicting the difficulty and response time of multiple-choice questions from the United States Medical Licensing Examination® (USMLE®). This exam is an important assessment for medical professionals. To predict these variables, we evaluated various BERT-like pre-trained transformer models. We combined these models with Scalar Mixing and two custom 2-layer classification heads, using learnable Rational Activations as the activation function. This multi-task setup allowed us to predict both item difficulty and response time. The results were noteworthy. Our models placed first out of 43 participants in predicting item difficulty and fifth out of 34 participants in predicting item response time. This demonstrates the potential of advanced AI techniques in improving the evaluation processes of critical exams like the…

Apr112024 by Sebastian GombertNo Comments

Team led by Sebastian Gombert wins one of two tracks at BEA 2024 shared task on predicting Item Difficulty and Item Response Time

Artificial Intelligence, Assessment, Award, Computational Psychometrics, Conference, Higher Education, New Pub, Workshop

For standardized exams to be fair and reliable, they must include a diverse range of question difficulties to accurately assess test taker abilities. Additionally, it's crucial to balance the time allotted per question to avoid making the test unnecessarily rushed or sluggish. The goal of this year's BEA shared task (competition) was to build systems which could predict Item Difficulty and Item Response Time for items taken from the United States Medical Licensing Examination (USMLE). EduTec member Sebastian Gombert designed systems which are able to predict both variables simultaneously. These placed first out of 43 for predicting Item Difficulty and fitfth out of 34 for predicting Item Response Time. They use modified versions of established transformer language models in a multitask setup. A corresponding system description paper titled Predicting Item…

Jan292024 by Sebastian GombertNo Comments

New Pub: From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study

Artificial Intelligence, Assessment, Computational Psychometrics, Empirical Study, Feedback, Higher Education, Journal, Publication, Special Issue, Technical paper

How can we give students highly informative feedback on their essays using natural language processing? In our new paper, led by Sebastian Gombert, we present a case study on using GBERT and T5 models to generate feedback for educational psychology students. In this paper: ➡ We implemented a two-step pipeline that segments the essays and predicts codes from the segments. The codes are used to generate feedback texts informing the students about the correctness of their solutions and the content areas they need to improve. ➡ We used 689 manually labelled essays as training data for our models. We compared GBERT, T5, and bag-of-words baselines for both steps. The results showed that the transformer-based models outperformed the baselines in both steps. ➡ We evaluated the feedback with a learner cohort…

Jun292023 by Sebastian GombertNo Comments

Workshop: Hyperchalk – How to implement Self-hosted Whiteboard Tasks @ JTEL Summer School 2023

Artificial Intelligence, Computer-supported collaborative learning, Learning Analytics, Summer School, Workshop

In this workshop which Lukas Menzel and I gave at the seventeenth JTEL Summer School, we explored the possibilities of our self-implemented whiteboard tool Hyperchalk. Hyperchalk is a backend for Excalidraw which allows for integrating learning management systems via LTI and collecting a complete history of trace data. After a short kick-off presentation, we let the participants design their own learning activities using the whiteboard. All participants created little tasks that other participants then solved. As most participants had a strong background in teaching, these were inspired by practical experiences. The tasks involved various topics, from stochastics to K12-level geography. At the end of the workshop, we taught the participants how to administrate the tool and how to set it up on their own servers. Overall, it was a successful…

Jun292023 by Sebastian GombertNo Comments

Workshop: Large Language Models for Feedback Generation @ JTEL Summer School 2023

Artificial Intelligence, Feedback, Summer School, Workshop

At the seventeenth JTEL Summer School, Lukas Menzel and I had the pleasure of giving a workshop on the potentials and pitfalls of large language models for generating learner feedback. We kicked the event off with a general presentation on large language models. We explained the technical properties of well-known language models such as BERT or GPT. Following this, we went into different setups that can be used for feedback generation. On the one hand, this can involve training a BERT-based model to predict codes for input responses that trigger OnTask-style feedback rules. While this approach is stable regarding what feedback students receive, it is also inflexible, as such feedback cannot necessarily mirror all detailed intricacies that might occur in a student's response. For this reason, it can feel kind…

Sep162022 by Sebastian GombertNo Comments

EC-TEL2022 best demo award goes to EduTec

Award, Conference

The demo paper Superpowers in the Classroom: Hyperchalk is an Online Whiteboard for Learning Analytics Data Collection and the corresponding tool Hyperchalk by Lukas Menzel, Sebastian Gombert, Daniele Di Mitri and Hendrik Drachsler were awarded the best demo award at the European Conference on Technology Enhanced Learning 2022. Hyperchalk allows hosting collaborative whiteboards which can be used to collaborate in real-time. It collects rich user data which can be used to conduct learning analytics. Hyperchalk was initially developed for the ALICE project.

Sep152022 by Sebastian GombertNo Comments

Sebastian Gombert @ EC-TEL 2022 Doctoral Consortium

Artificial Intelligence, Conference, Event, Learning Analytics, Research topic

At EC-TEL 2022, Sebastian Gombert presented his doctoral project "Methods and perspectives for the automated analytic assessment of free-text responses in formative scenarios". The goal behind this project is to build free-text assessment systems which can code constructed responses analytically. While much of the past work in response scoring focused on predicting holistic scores, analytic scoring allows to code free-text responses for multiple different aspects. The goal behind these efforts is to provide learners and teachers with useful feedback on each of them. A more detailed project overview will be released as part of the EC-TEL 2022 doctoral consortium proceedings soon.

Sep142022 by Sebastian GombertNo Comments

New Pub: Superpowers in the Classroom: Hyperchalk is an Online Whiteboard for Learning Analytics Data Collection

Computer-supported collaborative learning, Conference, Conference, Event, Further Education, General education, Higher Education, Learning Analytics, Lifelong Learning, Publication, School, Technical paper

A new system demonstration paper authored by Lukas Menzel, Sebastian Gombert, Daniele Di Mitri and Henrik Drachsler has been released as part of the ECTEL 2022 proceedings. In this paper, we present Hyperchalk, a self-hosted collaborative online whiteboard software. Similar to commercial solutions like Miro or Flinga, this software provides users with collaborative boards which they can use to draw, write or sketch together. However, unlike commercial solutions, Hyperchalk allows for collecting rich log data, which can be used to study the behaviour of its users and to allow Learning Analytics and studies on computer-supported collaborative learning. Moreover, Hyperchalk comes with a built-in replay mode which allows watching how users behave in its spaces. It supports the LTI1.3 standard, which enables seamless integration with learning management systems such as Moodle,…

Jan252022 by Sebastian GombertNo Comments

Team of EduTec and TBA Members Wins Runners-Up Prize @ NAEP Reading Automated Scoring Challenge

Artificial Intelligence, Assessment, Award, Computational Psychometrics, School

The NAEP Reading Automated Scoring Challenge was a competition which was held in late 2021 by the National Center for Education (USA). The goal was to evaluate the applicability of natural language processing methodology to the task of scoring of large scale constructed response assessment data. In addition to this, participating models were also required to be interpretable and free of algorithmic bias against different student demographics. The data set which was used to evaluate the systems consisted of responses from 4th and 8th grade school student to 20 different reading comprehension tasks. More than a dozen different teams from universities and private assessment companies submitted contributions. Out of these, 12 contributions, including ours, were selected as eligible with respect to interpretability and algorithmic fairness. These contributions were then ranked…