No document available.
Abstract :
[en] Educational large-scale assessments aim to evaluate school systems’ effectiveness by typically looking at aggregated levels of students’ performance. The developed assessment tools or tests are not intended or optimized to be used for diagnostic purposes on an individual level. In most cases, the underlying theoretical framework is based on national curricula and therefore too blurry for diagnostic test construction, and test length is too short to draw reliable inferences on individual level. This lack of individual information is often unsatisfying, especially for participating students and teachers who invest a considerable amount of time and effort, not to speak about the tremendous organizational work needed to realize such assessments. The question remains, if the evaluation could not be used in an optimized way to offer more differentiated information on students’ specific skills.
The present study explores the potential of Diagnostic Classification Models (DCM) in this regard, since they offer crucial information for policy makers, educators, and students themselves. Instead of a ranking of, e.g., an overall mathematics ability, student mastery profiles of subskills are identified in DCM, providing a rich base for further targeted interventions and instruction (Rupp, Templin & Henson, 2010; von Davier, M., & Lee, Y. S., 2019). A prerequisite for applying such models is well-developed, and cognitively described items that map the assessed ability on a fine-grained level. In the present study, we drew on 104 items that were developed on base of detailed cognitive item models for basic Grade 1 competencies, such as counting, as well as decomposition and addition with low numbers and high numbers (Fuson, 1988, Fritz & Ricken, 2008, Krajewski & Schneider, 2009). Those items were spread over a main test plus 6 different test booklets and administered to a total of 5963 first graders within the Luxembourgish national school monitoring Épreuves standardisées.
Results of this pilot study are highly promising, giving information about different student’s behaviors patterns: The final DCM was able to distinguish between different developmental stages in the domain of numbers & operations, on group, as well as on individual level. Whereas roughly 14% of students didn’t master any of the assessed competencies, 34% of students mastered all of them including addition with high numbers. The remaining 52% achieved different stages of competency development, 8% of students are classified only mastering counting, 15% of students also can master addition with low numbers, meanwhile 20% of students additionally can master decomposition, all these patterns reflect developmental models of children’s counting and concept of numbers (Fritz & Ricken, 2008; see also Braeuning et al, 2021). Information that could potentially be used to substantially enhance large-scale assessment feedback and to offer further guidance for teachers on what to focus when teaching. To conclude, the present results make a convincing case that using fine-grained cognitive models for item development and applying DCMs that are able to statistically capture these nuances in student response behavior might be worth the (substantially) increased effort.
References:
Braeuning, D. et al (2021)., Long-term relevance and interrelation of symbolic and non-symbolic abilities in mathematical-numerical development: Evidence from large-scale assessment data. Cognitive Development, 58, https://doi.org/10.1016/j.cogdev.2021.101008.
Fritz, A., & Ricken, G. (2008). Rechenschwäche. utb GmbH.
Fuson, K. C. (1988). Children's counting and concepts of number. Springer-Verlag Publishing.
Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guildford Press.
Von Davier, M., & Lee, Y. S. (2019). Handbook of diagnostic classification models. Cham: Springer International Publishing.
