autoMATH: automatic mathematic item generation purposefully not powered by AI

Despite decades of research on model-based automatic item generation (e.g. Gierl et al., 2023), the search for a solution that will consistently produce items that are psychometrically sound is still on. Comparing the traditional method of developing test items, which is time-consuming and costly (Kosh et al., 2018), with items created by (agnostic) generative AI (Laverghetta & Licat, 2023), the latter appears to be the silver bullet for automatic item production. Though cost-effective, these AI generated items jeopardize construct validity standards since they lack traceability of item components (such as the stem, question, and distractors). In this presentation, we demonstrate and discuss auto.MATH, an automatic item generator based on cognitive models for mathematical competencies that have been psychometrically evaluated, in order to argue against discarding model-based automatic item creation too soon. These models were developed on base of the large, multilingual item pools of Luxembourg’s national school monitoring program (Épreuves Standardisées) which uses the national education curriculum as guidance for item development. The theoretical framework of the models, which is based on data that has been psychometrically tested and allows for the distinction of different difficulty levels, is a key characteristic that sets auto.MATH apart from other programs, particularly AI-based models. This means that everything that is entered, from the process of creation to the final product and its characteristics, is founded on theory and can be traced. We‘ll discuss areas of application, such as students‘ training needs, especially in areas where national school monitoring programs have identified shortcomings.