Generative item models as learning opportunity: a psychometric analysis of 24 item University of Luxembourg models in the mathematical domain of space & form

Despite the disruptive impact of generative AI on item writing, there are still merits to pre-defined,
template-like item models that can be used in a generative way too. These models offer transparency, as they are not a black box and can be validated by subject matter experts. They also allow for predictable characteristics in generated items and facilitate systematic exploration of the models' psychometric properties, yielding valuable insights into the targeted construct.
Focusing on the mathematical domain of space and form, this study psychometrically analyzes 25
cognitive item models developed by a team of national experts. These models were used to create items
for Grades 1, 3, and 5 in the Luxembourgish school system. Each model underwent administration in six experimentally varied versions to assess the impact of contextual presentation and problem
characteristics identified by cognitive psychology as influential in problem-solving processes.
Data from the annual school monitoring Épreuves standardisées for Grade 1 (n = 3694), Grade 3 (n =
4625), and Grade 5 (n = 3716) was analyzed in-depth by descriptive comparisons of resulting IRT
parameters, and the estimation of manipulated problem characteristics’ impact on item difficulty by using a generalized linear mixed model with students as random effect (GLMM, De Boeck et al., 2011).
This allowed for an evaluation of the stability, predictability, and unbiased nature of the psychometric characteristics of items generated by each model, particularly concerning subgroups known to be disadvantaged in the Luxembourgish school system.
The findings offer significant insights into the space and form domain in mathematics, a vital but less studied area. They reveal how the graphical nature of items in this domain is substantially influenced by their presentation, underscoring the importance of controlled, template-based item generation for ensuring versatility and fairness.