Assessment Archives - EduTec Science

Oct142024 by Heather WehnerNo Comments

New Pub: Conceptual Map Assessment Through Structure Classification

Assessment, Conference, Feedback, Publication, School

In educational settings concept maps are often chosen as a tool to help knowledge constructions be visualized. While mapping out concepts and their relationships, students are able to show how well (or not well) they understand certain subjects. A newly published study, which was presented at the AIED 2024 in Recife, Brazil, takes their usage a step further, examining the structural patterns in concept maps and developing an automated system to classify them. The researchers categorized 511 different concept maps into three key structures: spoke, network and chain. Each structure type can provide its own insight about the deepness of students’ understanding. For example, a "spoke" structure may indicate a surface-level understanding, while a "network" might reflect a more complex comprehension. The researchers trained multiclass classification models using statistical data…

Jul182024 by Heather WehnerNo Comments

New Pub: Revolutionizing Essay Scoring with Hierarchical Rater Models

Artificial Intelligence, Assessment, Higher Education, Journal, New Pub

For a special issue on Natural Language Processing in Psychology we proposed a hierarchical rater model-based approach to address the challenges in automatic essay scoring. Essay writing tests are an integral part of educational systems, essential for assessing students' critical thinking, articulation and understanding. Since the manual scoring process requires significant resources and time, teachers are beginning to use Automated Essay Scoring (AES), which is potentially capable of alleviating the manual effort involved. #AutomatedEssayScoring #NaturalLanguageProcessing #FormativeAssessment #EducationalTechnology #MachineLearning #AIinEducation #HierarchicalRaterModel #EdTech #ScoringAutomation #AI #AssessmentTools #MeasurementInvariance #TransformerModels #EducationResearch #UniversityTesting #AIModels #TechInEducation There are an abundance of available models and each one has its own unique features and scoring methods. Thus, selecting the optimal model is complex and challenging, especially when different aspects of content have to be assessed over a number…

Jul12024 by Sebastian GombertNo Comments

New pub: Predicting Item Difficulty and Item Response Time with Scalar-mixed Transformer Encoder Models and Rational Network Regression Heads

Artificial Intelligence, Assessment, Computational Psychometrics, Conference, Higher Education, Publication, Workshop

In a contribution to the BEA 2024 Shared Task, we addressed the challenge of predicting the difficulty and response time of multiple-choice questions from the United States Medical Licensing Examination® (USMLE®). This exam is an important assessment for medical professionals. To predict these variables, we evaluated various BERT-like pre-trained transformer models. We combined these models with Scalar Mixing and two custom 2-layer classification heads, using learnable Rational Activations as the activation function. This multi-task setup allowed us to predict both item difficulty and response time. The results were noteworthy. Our models placed first out of 43 participants in predicting item difficulty and fifth out of 34 participants in predicting item response time. This demonstrates the potential of advanced AI techniques in improving the evaluation processes of critical exams like the…

May242024 by Heather WehnerNo Comments

FoLA supports Simulation-based Training for High-risk Clinical Situations

Assessment, Empirical Study, Feedback, Further Education, Higher Education, Learning Analytics, Learning Design, medical education, Multimodal Learning Analytics, Workshop

From May 16th to 17th, 2024, Hendrik Drachsler and Marcel Schmitz (TETRA AI, ZUYD University of Applied Science) had the pleasure of providing a FoLA workshop at the College of Anaesthesiologists of Ireland (CAI) in Dublin, Ireland, under the support of the Insight research centre. Together with members from CAI and from the ASSERT Centre, College of Medicine and Health, University College Cork (UCC), they used the FoLA tool to plan a simulation-based training for high-risk clinical situations using highly informative feedback. Data from these trainings will be collected and analyzed to determine how effective this training is in improving the performance of the attendees. The study is being planned to take place with around 100 doctors in training at two simulation training centers: at ASSERT Centre UCC and at…

Apr112024 by Sebastian GombertNo Comments

Team led by Sebastian Gombert wins one of two tracks at BEA 2024 shared task on predicting Item Difficulty and Item Response Time

Artificial Intelligence, Assessment, Award, Computational Psychometrics, Conference, Higher Education, New Pub, Workshop

For standardized exams to be fair and reliable, they must include a diverse range of question difficulties to accurately assess test taker abilities. Additionally, it's crucial to balance the time allotted per question to avoid making the test unnecessarily rushed or sluggish. The goal of this year's BEA shared task (competition) was to build systems which could predict Item Difficulty and Item Response Time for items taken from the United States Medical Licensing Examination (USMLE). EduTec member Sebastian Gombert designed systems which are able to predict both variables simultaneously. These placed first out of 43 for predicting Item Difficulty and fitfth out of 34 for predicting Item Response Time. They use modified versions of established transformer language models in a multitask setup. A corresponding system description paper titled Predicting Item…

Mar272024 by Heather WehnerNo Comments

CORE Roundtable in Munich

Assessment, Critical Online Reasoning, Multimodal Learning Analytics, Project, Project meeting

Hendrik Drachsler, Sebastian Gombert and Gianluca Romano participated at the Roundtable in Munich for the CORE project (Critical Online Reasoning in Higher Education) from 04.03.-05.03.2024. In those two days, our team had the chance to recapitulate on how the infrastructure stood strong during the first survey from December 2023 to February 2024, and pave the way for next steps and surveys. In summary, the infrastructure performed well. It dealt with approximately 10Mio. requests per seconds and the majority of hurdles participants reported were out of our authority. Requests from participants were dealt with quickly in a few days on average. Even though we are proud of our achievements there is still a lot to be done for future surveys. For the agenda we split into smaller groups, each of us…

Mar152024 by Heather WehnerNo Comments

New Pub: AI program doesn’t make kids better at math, but it makes them more independent

Assessment, Journal, Publication, School

Students who receive math tutoring from an artificial intelligence (AI) program perform no better than students who are taught by a "real" teacher. These students do, however, need less help learning. This is the conclusion of Rashmi Khazanchi from the Open University of the Netherlands together with Hendrik Drachsler and Daniele Di Mitri. Math Lessons with AI The researchers examined the effectiveness of the Assessment and Learning in Knowledge Spaces (ALEKS) tutoring program, called Intelligent Tutoring Systems (ITS). Previous studies have shown that students learn math better using software than traditional teaching methods. Previous studies on ALEKS have also shown that, thanks to this program, students memorize more knowledge, perform better, experience more engagement in mathematics and drop out less. The advantage of an ITS like ALEKS is that it…

Jan292024 by Sebastian GombertNo Comments

New Pub: From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study

Artificial Intelligence, Assessment, Computational Psychometrics, Empirical Study, Feedback, Higher Education, Journal, Publication, Special Issue, Technical paper

How can we give students highly informative feedback on their essays using natural language processing? In our new paper, led by Sebastian Gombert, we present a case study on using GBERT and T5 models to generate feedback for educational psychology students. In this paper: ➡ We implemented a two-step pipeline that segments the essays and predicts codes from the segments. The codes are used to generate feedback texts informing the students about the correctness of their solutions and the content areas they need to improve. ➡ We used 689 manually labelled essays as training data for our models. We compared GBERT, T5, and bag-of-words baselines for both steps. The results showed that the transformer-based models outperformed the baselines in both steps. ➡ We evaluated the feedback with a learner cohort…

Dec42023 by Prof. Dr. Hendrik DrachslerNo Comments

16th eLearning Netzwerktag: An Insightful Recap of the Fast-Paced Year

Artificial Intelligence, Assessment, Augmented Reality, Competence development, Computational Psychometrics, Computer-supported collaborative learning, Conference, Event, Feedback, Gender, Higher Education, Learning Analytics, Learning Design

The annual eLearning Netzwerktag was a highly anticipated one-day event where the eLearning community of Frankfurt and the surrounding areas gathered to present the highlights of the past year to the public. On November 21, 2023, the event took place at Campus Westend, Goethe University Frankfurt am Main. Among the speakers, the Prof. Dr. Maren Scheffel, Prof. Dr. Franziska Matthäus , CIO of Goethe University Ulrich Schielein, Prof. Dr.Hendrik Drachsler, Director of studiumdigitale, delivered an opening speech that reflected on an incredible year, with a particular focus on the advancements in generative Artificial Intelligence applications. Hendrik Drachsler's speech highlighted the significant developments in the field of digital learning. At the previous Netzwerktag, applications like ChatGPT, Midjourney, Stablediffusions, and open language models (LLMs) such as LAMA were relatively unknown to most…

Dec12023 by Heather WehnerNo Comments

New pub: Measuring Efficacy of ALEKS as a Supportive Instructional Tool in K-12 Math Classroom with Underachieving Students

Assessment, Journal, New Pub, Publication, School

In a recent quasi-experimental research study, the effectiveness of Assessment and Learning in Knowledge Spaces (ALEKS), an Intelligent Tutoring System (ITS), took center stage in the realm of 8th-grade mathematics education. The study aimed to determine whether ALEKS could bring a statistically significant improvement in students' mathematics achievement compared to traditional teacher-led instructions. The research involved 158 8th-grade students categorized as 'underachieving students, with 60 in the teacher-led group and 98 in the ALEKS-led group. The study used a non-randomized approach to compare the outcomes of teacher-led instructions to ALEKS-led instructions over two consecutive years. In the first year McGraw's curriculum "Reveal" was used exclusively without ALEKS. In the second year ALEKS was incorporated as a supplemental tool in a math support class. The study incorporated a rigorous methodology, utilizing…