Assessment Archives - EduTec Science

Apr112024 by Sebastian GombertNo Comments

Team led by Sebastian Gombert wins one of two tracks at BEA 2024 shared task on predicting Item Difficulty and Item Response Time

Artificial Intelligence, Assessment, Award, Computational Psychometrics, Conference, Higher Education, New Pub, Workshop

For standardized exams to be fair and reliable, they must include a diverse range of question difficulties to accurately assess test taker abilities. Additionally, it's crucial to balance the time allotted per question to avoid making the test unnecessarily rushed or sluggish. The goal of this year's BEA shared task (competition) was to build systems which could predict Item Difficulty and Item Response Time for items taken from the United States Medical Licensing Examination (USMLE). EduTec member Sebastian Gombert designed systems which are able to predict both variables simultaneously. These placed first out of 43 for predicting Item Difficulty and fitfth out of 34 for predicting Item Response Time. They use modified versions of established transformer language models in a multitask setup. A corresponding system description paper titled Predicting Item…

Mar272024 by Heather WehnerNo Comments

CORE Roundtable in Munich

Assessment, Critical Online Reasoning, Multimodal Learning Analytics, Project, Project meeting

Hendrik Drachsler, Sebastian Gombert and Gianluca Romano participated at the Roundtable in Munich for the CORE project (Critical Online Reasoning in Higher Education) from 04.03.-05.03.2024. In those two days, our team had the chance to recapitulate on how the infrastructure stood strong during the first survey from December 2023 to February 2024, and pave the way for next steps and surveys. In summary, the infrastructure performed well. It dealt with approximately 10Mio. requests per seconds and the majority of hurdles participants reported were out of our authority. Requests from participants were dealt with quickly in a few days on average. Even though we are proud of our achievements there is still a lot to be done for future surveys. For the agenda we split into smaller groups, each of us…

Mar152024 by Heather WehnerNo Comments

New Pub: AI program doesn’t make kids better at math, but it makes them more independent

Assessment, Journal, Publication, School

Students who receive math tutoring from an artificial intelligence (AI) program perform no better than students who are taught by a "real" teacher. These students do, however, need less help learning. This is the conclusion of Rashmi Khazanchi from the Open University of the Netherlands together with Hendrik Drachsler and Daniele Di Mitri. Math Lessons with AI The researchers examined the effectiveness of the Assessment and Learning in Knowledge Spaces (ALEKS) tutoring program, called Intelligent Tutoring Systems (ITS). Previous studies have shown that students learn math better using software than traditional teaching methods. Previous studies on ALEKS have also shown that, thanks to this program, students memorize more knowledge, perform better, experience more engagement in mathematics and drop out less. The advantage of an ITS like ALEKS is that it…

Jan292024 by Sebastian GombertNo Comments

New Pub: From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study

Artificial Intelligence, Assessment, Computational Psychometrics, Empirical Study, Feedback, Higher Education, Journal, Publication, Special Issue, Technical paper

How can we give students highly informative feedback on their essays using natural language processing? In our new paper, led by Sebastian Gombert, we present a case study on using GBERT and T5 models to generate feedback for educational psychology students. In this paper: ➡ We implemented a two-step pipeline that segments the essays and predicts codes from the segments. The codes are used to generate feedback texts informing the students about the correctness of their solutions and the content areas they need to improve. ➡ We used 689 manually labelled essays as training data for our models. We compared GBERT, T5, and bag-of-words baselines for both steps. The results showed that the transformer-based models outperformed the baselines in both steps. ➡ We evaluated the feedback with a learner cohort…

Dec42023 by Prof. Dr. Hendrik DrachslerNo Comments

16th eLearning Netzwerktag: An Insightful Recap of the Fast-Paced Year

Artificial Intelligence, Assessment, Augmented Reality, Competence development, Computational Psychometrics, Computer-supported collaborative learning, Conference, Event, Feedback, Gender, Higher Education, Learning Analytics, Learning Design

The annual eLearning Netzwerktag was a highly anticipated one-day event where the eLearning community of Frankfurt and the surrounding areas gathered to present the highlights of the past year to the public. On November 21, 2023, the event took place at Campus Westend, Goethe University Frankfurt am Main. Among the speakers, the Prof. Dr. Maren Scheffel, Prof. Dr. Franziska Matthäus , CIO of Goethe University Ulrich Schielein, Prof. Dr.Hendrik Drachsler, Director of studiumdigitale, delivered an opening speech that reflected on an incredible year, with a particular focus on the advancements in generative Artificial Intelligence applications. Hendrik Drachsler's speech highlighted the significant developments in the field of digital learning. At the previous Netzwerktag, applications like ChatGPT, Midjourney, Stablediffusions, and open language models (LLMs) such as LAMA were relatively unknown to most…

Dec12023 by Heather WehnerNo Comments

New pub: Measuring Efficacy of ALEKS as a Supportive Instructional Tool in K-12 Math Classroom with Underachieving Students

Assessment, Journal, New Pub, Publication, School

In a recent quasi-experimental research study, the effectiveness of Assessment and Learning in Knowledge Spaces (ALEKS), an Intelligent Tutoring System (ITS), took center stage in the realm of 8th-grade mathematics education. The study aimed to determine whether ALEKS could bring a statistically significant improvement in students' mathematics achievement compared to traditional teacher-led instructions. The research involved 158 8th-grade students categorized as 'underachieving students, with 60 in the teacher-led group and 98 in the ALEKS-led group. The study used a non-randomized approach to compare the outcomes of teacher-led instructions to ALEKS-led instructions over two consecutive years. In the first year McGraw's curriculum "Reveal" was used exclusively without ALEKS. In the second year ALEKS was incorporated as a supplemental tool in a math support class. The study incorporated a rigorous methodology, utilizing…

Jun22023 by Heather WehnerNo Comments

New pub: Digitalisierung und Diagnostik in Schulen – Herausforderungen für Bildungspraxis und Bildungsforschung

Assessment, Book chapter, Computational Psychometrics, Digitalisation, Learning Analytics, Research Methods, Research topic, School

In the spring of 2020, schools faced an unprecedented challenge as the pandemic disrupted traditional modes of instruction and school development. With on-site learning replaced by digital formats and distance communication, educators had to quickly adapt to the new normal. Amidst these changes, the field of education encountered specific challenges related to digital school management, digital learning, and assessing learning progress. Particularly, computer-aided diagnostics emerged as a valuable tool for gaining insights not only into learning outcomes but also into the learning process itself. Researchers became intrigued by the potential of digital media in shaping learning experiences and how the resulting data could be effectively utilized in educational practice. This paper explores the current challenges and potentials in computer-based, learning-accompanying diagnostics. The primary hurdles involve implementing suitable instruments within schools…

May152023 by Prof. Dr. Hendrik DrachslerNo Comments

New research program: Towards Highly Informative Learning Analytics

Artificial Intelligence, Assessment, Book, Computational Psychometrics, Computer-supported collaborative learning, Feedback, Further Education, Higher Education, Keynote, Learning Analytics, Learning Design, Research Methods, School

On May 12, 2023, the Highly Informative Learning Analytics (HILA) research program of the EduTec@DIPF, studiumdigitale@Goethe University Frankfurt and the Open Learning and Instruction group@Open Universiteit was presented by Hendrik Drachsler at the main campus of the Open University of the Netherlands. The release of the HILA research program marks a significant milestone for the collaboration in the field of learning analytics between of the Dutch-German research collective. The HILA research program is focused on developing new tools and methods to collect, analyze, and interpret data that can help educational institutions to understand the learning process better. As part of the program's launch, a keynote by Ioana Jivet on student-facing learning analytics was provided. Ioana reported on two empirical studies investigating the effect of data-driven feedback on students. [pdf-embedder url="https://edutec.science/wp-content/uploads/2023/05/2023_05-Keynote-Symposium-Hendrik.pdf"…

Apr212023 by Heather WehnerNo Comments

CORE: New project investigates students’ critical use of digital media

Assessment, Digitalisation, Higher Education, Project, Research topic, Target group

Students are increasingly using information from the Internet to learn for their studies. According to recent surveys, they even cite the Internet as their main source when searching for learning materials. Research shows that students often rely on inappropriate, irrelevant and scientifically unsupported sources when searching the web. Little is known about exactly how self-directed learning takes place on the Internet and how information is selected and processed. This will now be addressed by the new research group "Kritisches Denken in Online-Lernumgebungen in der Hochschulbildung (CORE)” (Critical Thinking in Online Learning Environments in Higher Education), which is being funded by the German Research Foundation (DFG) with more than 5 million euros over the next four years. The CORE project will investigate the critical use of digital media and information among…

Feb222023 by Jane YauNo Comments

HIKOF – Evaluation Study Completed

Artificial Intelligence, Assessment, Feedback, Learning Analytics, Project

In HIKOF (highly informative feedback for digital learning) project, the evaluation study has been successfully completed, where feedback was provided for about 1000 students in a Teacher Education Course (from Psychology department). Students' feedback on the feedback that was given to them was also collected to determine how comprehensive our feedback was, if it motivated them, and was helpful for them, and enhanced their learning progress. Specifically, the evaluation study focuses on the use of machine learning for the creation of personalized feedback that is both formative and summative. Data from a pilot study serves as a starting point for the evaluation to further optimize the AI technology for generating feedback. The evaluation study aims to investigate the effect of Highly Informative Automated Feedback (HIAF) on student performance and perception…