AIG and student math assessment: Psychometric characteristics of automatically generated items

National school monitoring plans, like the Luxembourgish Épreuves
Standardisées (ÉpStan), always call for psychometrically tested items which are
costly and time-consuming (Gierl et. al., 2012). The use of models and templates to
automatically generate vast numbers of items is a method that is becoming
increasingly popular and has the potential to clench the thirst for more items year
after year. Mathematical evaluation frequently adopts a largely language free
approach in extremely heterogeneous environments like Luxembourg, employing
visuals rather than words to provide context for the mathematical problem
(Sonnleitner et al., 2018). Yet research on utilizing images in text items indicates
ambiguous results depending on their role and perception (Lindner et al. 2016;
Lindner 2020). It is thus not entirely clear if template-based items that are imagebased
share the same item characteristics.
The pretests of ÉpStan 2021 included model-based generated items. By drawing
on data of seventh graders, this study’s main aim is to analyze, in an explorative way,
the impact of construct-relevant (task or problem features) and construct-irrelevant
(semantic embedding) variations in items on their empirical difficulty and
psychometric characteristics. The ÉpStan quality standards, which are equivalent to
common IRT quality criteria (rit >.25; outfit >1.2), were met by all the created items.
Further examined and discussed will also be the impact of these variations on
subgroup level and what this could imply for the fairness of the automatically
generated items in this context.