Enhancing Large-Scale Mathematical Assessments: Integrating Cognitive Diagnostic Models with Hierarchical Attribute Structures for Improved Student Diagnostics

Cognitive diagnostic models (CDMs) provide a fine-grained analysis of students’ cognitive abilities by determining their mastery or non-mastery of specific attributes. CDMs have been retrofitted to existing non-diagnostic (inter)national large-scale standardized mathematics assessments. However, in the absence of a cognitive development framework for constructing test items in retrofitting studies, an inferred substantive model and a Q matrix are typically constructed to define the attributes measured in the test. This indirect approach can result in information loss, less precise modeling of cognitive attributes, and inaccurate student classifications. This study uses a hierarchical cognitive diagnostic model (HCDM) to explore the feasibility of incorporating cognitive models into the development of large scale assessment items that inherently generate Q-matrices for CDM analysis. It examines whether integrating theoretical assumptions about hierarchical relationships between mathematical attributes can improve the accuracy and effectiveness of the model. In contrast to previous CDM studies, an eight-attribute Q-matrix was systematically designed. Items were constructed based on the Q-matrix, which was derived from curriculum, cognitive models, and specified attribute hierarchies. The test was administered to 5,336 third-grade students in Luxembourg. The HCDM was evaluated against the G-DINA model. The results showed that: (1) the mastery rates derived from the HCDM align more closely with the developmental progression of mathematical ability, supporting the notion that theoretically grounded Q-matrix design yields more interpretable latent classes; (2) incorporating hierarchical attribute relationships enhances the effectiveness of diagnostic assessments by generating more meaningful and accountable classifications of student proficiency; and (3) the HCDM aligns more closely with didactic theories of mathematical development, because it better captures the structured progression of learning, where mastery of foundational skills is necessary for acquiring more complex competencies. This study underscores the importance of integrating cognitive models into assessment design and highlights the advantages of using HCDMs to improve large-scale educational diagnostics.