Benefits of Using Cognitive Models Within a Mathematics Large-Scale Assessment

Scientific Conference (2023, April 13)

For several decades, researchers have suggested cognitive models as superior basis for item development (Hornke & Habon, 1986; Leighton & Gierl, 2011). Such models would make item writing decisions ... [more ▼]

For several decades, researchers have suggested cognitive models as superior basis for item development (Hornke & Habon, 1986; Leighton & Gierl, 2011). Such models would make item writing decisions explicit and therefore more valid. By further formalizing such models, even automated item generation with its manifold advantages for economic test construction, and increased test security is possible. If item characteristics are stable, test equating would be rendered unnecessary allowing for individual but equal tests, or even adaptive or multistage testing without extensive pre-calibration. Finally, validated cognitive models would allow for applying Diagnostic Classification Models that provide fine-grained feedback on students’ competencies (Leighton & Gierl, 2007; Rupp, Templin, & Henson, 2010). Remarkably, despite constantly growing need for validated items, educational large-scale assessments (LSAs) have largely forgone cognitive models as template for item writing. Traditional, often inefficient item writing techniques prevail and participating students are offered a global competency score at best. This may have many reasons, above all the focus of LSAs on the system and not individual level. Many domains lack the amount of cognitive research necessary for model development (e.g. Leighton & Gierl, 2011) and test frameworks are mostly based on didactical viewpoints. Moreover, developing an empirically validated cognitive model remains a challenge. Considering the often time-sensitive test development cycles of LSAs, the balance clearly goes against the use of cognitive models. Educational LSAs are meant to stay, however, and the question remains, whether increased effort and research on this topic might pay off in the long run by leveraging all benefits cognitive models have to offer. In total, 35 cognitive item models were developed by a team of national subject matter experts and then used for algorithmically producing items for the mathematical domain of numbers & shapes. Each item model was administered in 6 experimentally varied versions to investigate the impact of problem characteristics which cognitive psychology identified to influence the problem-solving process. Based on samples from Grade 1 (n = 5963), Grade 3 (n = 5527), Grade 5 (n = 5291), and Grade 7 (n = 3018), this design allowed for evaluating whether psychometric characteristics of produced items per model are stable, and can be predicted by problem characteristics. After item calibration (1-PL model), each cognitive model was analyzed in-depth by descriptive comparisons of resulting IRT parameters, and using the LLTM (Fischer, 1973). In a second step, the same items were analyzed using the G-DINA model (Torre & Minchen, 2019) to derive classes of students for the tested subskills. The cognitive models served as basis for the Q-matrix necessary for applying the diagnostic measurement model. Results make a convincing case for investing the (substantially) increased effort to base item development on fine-grained cognitive models. Model-based manipulations of item characteristics were largely stable and behaved according to previous findings in the literature. Thus, differences in item difficulty could be shaped and were stable over different administrations. This remained true for all investigated grades. The final diagnostic classification models distinguished between different developmental stages in the domain of numbers & operations, on group, as well as on individual level. Although not all competencies might be backed up by literature from cognitive psychology yet, our findings encourage a more exploratory model building approach given the usual long-term perspective of LSAs. [less ▲]

The formation of academic self-concept and interest in primary school: Examining the generalized internal/external frame of reference model with first- and third-grade children

in Contemporary Educational Psychology (2023)

Findings from the ÉpStan National Education Monitoring against the Background of the COVID-19 Pandemic

E-print/Working paper (2022)

Subjektives Wohlbefinden in der 5. und 9. Schulklasse: gibt es einen Zusammenhang mit dem Bildungsweg und der schulischen Leistung?

Scientific Conference (2022, March 09)

Der Bildungsauftrag unserer heutigen Wissensgesellschaft vereint ein vielseitiges Spektrum an Kompetenzen, die den Schüler*innen vermittelt werden sollen. Die Lernziele beinhalten nicht nur akademischen ... [more ▼]

Der Bildungsauftrag unserer heutigen Wissensgesellschaft vereint ein vielseitiges Spektrum an Kompetenzen, die den Schüler*innen vermittelt werden sollen. Die Lernziele beinhalten nicht nur akademischen Erfolg, sondern auch schulisches Wohlbefinden. In der Bildungsforschung haben affektive und sozio-emotionale Faktoren sowie deren Einfluss auf das Erreichen von Lernzielen über die letzten Jahrzehnte an Interesse gewonnen (s.a. Hascher et al., 2018). Subjektives Wohlbefinden (SWB) ist ein komplexes, multidimensionales Phänomen, welches emotionale, soziale und kognitive Facetten umfasst (Hascher & Edlinger, 2009). Das SWB wird als Grundlage für erfolgreiches Lernen betrachtet (Hascher & Hagenauer, 2011), wobei der Zusammenhang je nach Entwicklungsstadium der Schüler*innen variieren kann. Ergebnisse einer Metaanalyse (Bücker et al., 2018) zeigten eine statistisch signifikante mittlere Effektstärke für den Zusammenhang zwischen SWB und Leistung, wobei diese Ergebnisse über verschiedene Ebenen soziodemografischer Merkmale, SWB-Domäne und Indikatoren der Leistung hinweg stabil waren. Außerdem zeigten Gutman und Voraus (2012) in einer längsschnittlichen Studie mit einer Kohorte von Schüler*innen zwischen 7 und 13 Jahren, schwache bis mittlere Korrelationen zwischen unterschiedlichen Dimensionen des Wohlbefindens und aktueller sowie späterer akademischer Leistung. In dieser Studie haben wir den Zusammenhang zwischen verschiedenen Dimensionen des SWB und standardisierten Kompetenztestergebnissen zu verschiedenen Zeitpunkten (5. und 9. Schulkasse) untersucht. Ein erstes Ziel bestand darin, die Unterschiede des Wohlbefindens in Bezug auf das Entwicklungsstadium zu untersuchen, wobei wir auch den Einfluss von Klassenwiederholung und Schulzweig betrachteten. Ein weiteres Ziel der Studie bestand darin, den Zusammenhang zwischen SWB und Leistung unter Berücksichtigung sozio-demografischer Variablen zu ermitteln. Die Ergebnisse basieren auf den Daten der gesamte Kohorte von Fünft- und Neuntklässler*innen (N=5159 bzw. N=6279), die im Rahmen des nationalen Schulmonitoring (Luxembourg School Monitoring Programm „Épreuves Standardisées“; Martin et al., 2015) im November 2018 in Luxemburg erhoben wurden. Im Rahmen dieser Erhebung wurden sowohl standardisierte Schulleistungstests als auch ein Fragebogen zu soziodemographischen und sozio-emotionalen Aspekten durchgeführt. Vier Domäne des SWB wurden erfasst: Selbstkonzept, Schulangst, soziale- sowie emotionale Inklusion. Die standardisierten Leistungstests umfassten Leseverstehen in Deutsch und Französisch sowie Mathematik. Zusätzlich wurden über einen Schüler- oder Elternfragebogen weitere sozio-demographische Merkmale erfasst. Der Zusammenhang zwischen SWB und Entwicklungsstadium (Schulklasse) unter Einbeziehung von Klassenwiederholung und Schulzweig wurde mittels zwei mixed model Analysen überprüft. Die Ergebnisse zeigten, dass Schüler*innen in der 5. Klasse höhere Werte von SWB angaben als Schüler*innen in der 9. Klasse, F(8,121164)=180.61, p<.001. Zusätzlich wurde das SWB negativ beeinflusst durch Klassenwiederholung, F(8, 63989)=17.75, p<.001. Neuntklässler*innen in anspruchsvolleren Schulzweigen gaben höhere Werte von SWB an als Schüler*innen in niedrigeren Schulzweigen, F(2,40219)=15.71, p<001. Die Schulleistung wurde über eine schrittweise Regression vorhergesagt: zunächst wurden sozio-demographische Hintergrundvariable (Geschlecht, Migrationshintergrund, HISEI der Eltern) dem Model hinzugefügt und, in einem zweiten Schritt, Indikatoren des SWB. Die Ergebnisse zeigten, dass in der 5. Klasse 13% und in der 9. Klasse 19% der Varianz in der Schulleistung durch soziodemografische Variablen vorausgesagt werden kann. Sowohl für Fünft- als auch für Neuntklässler*innen, erklärten die Dimensionen des SWB zusätzliche 6% bzw. 4% der Varianz. Die Ergebnisse dieser Studie zeigten, dass Entwicklungsstadium, Klassenwiederholung und Schulzweig einen Einfluß auf das SWB der Schüler*innen haben. Darüber hinaus zeigten die Ergebnisse, dass das SWB über soziodemografische Merkmale hinaus zur Erklärung der schulischen Leistung beiträgt. In Anbetracht ihrer Ergebnisse, kann diese Studie auch die Diskussion um Klassenwiederholung als pädagogische Intervention und um die Praxis der Aufgliederung von Schüler*innen nach Leistungsniveau innerhalb und zwischen Schulformen bereichern. Während sich die meisten Studien zu den Effekten dieser Interventionen auf die schulische Leistung konzentrierten, zeigt die vorliegende Studie, dass diese Maßnahmen auch das SWB betreffen. Weitere (längsschnittliche) Studien könnten darauf eingehen, inwieweit es letztendlich zu einem kumulativen Effekt auf die schulische Leistung kommen kann oder ob und inwiefern das SWB den Zusammenhang zwischen diesen Faktoren und der schulischen Leistung beeinflussen kann. [less ▲]

Academic Achievement and Subjective Well-being: A Representative Cross-sectional Study

in Heinen, Andreas; Samuel, Robin; Vögele, Claus (Eds.) et al Wohlbefinden und Gesundheit im Jugendalter (2022)

Formal education is a very important, time-intensive, and highly consequential aspect of adolescents’ everyday life. School as well as education can influence adolescents’ well-being in both the short ... [more ▼]

Formal education is a very important, time-intensive, and highly consequential aspect of adolescents’ everyday life. School as well as education can influence adolescents’ well-being in both the short- and long-term. In return, adolescents’ well-being in- and outside school may affect their educational achievement. The objective of the present study is to investigate how self-reported dimensions of adolescents’ subjective well-being (SWB) in an educational context (i.e., academic self-concept, school anxiety, social and emotional inclusion) relate to educational pathways (regular vs. irregular school transitions; attendance of more vs. less prestigious secondary school tracks) and standardized assessment scores in key academic areas (i.e., mathematics and languages). Drawing on representative data emerging from the Luxembourg School Monitoring Programme “Épreuves Standardisées” (academic year 2018/2019), the relationship between academic achievement and students’ self-reported well-being was analysed cross-sectionally for the entire student cohorts of 5th and 9th graders. Result indicated that grades and educational pathways affect SWB, whereby in general lower ratings of SWB were observed in older students, students that experienced grade retention and students in lower secondary school tracks. Furthermore, ratings of SWB explained a significant proportion of variance in academic achievement in bot Grade 5 and Grade 9. These findings highlight the importance of student´ SWB in education. SWB may not only affect academic achievement, but also impact motivation and engagement and hence long-term educational success. Implications of the findings for research and educational debate are discussed. [less ▲]

Nouveaux résultats longitudinaux issus du monitoring scolaire national ÉpStan en première et troisième année scolaire (cycles 2.1 et 3.1) : tendance négative au niveau du développement des compétences et redoublements inefficaces

in LUCET; SCRIPT (Eds.) Rapport national sur l’éducation au Luxembourg 2021 (2021)

Neue längsschnittliche Befunde aus dem nationalen Bildungsmonitoring ÉpStan in der 3. und 9. Klasse: Schlechtere Ergebnisse und wirkungslose Klassenwiederholungen

in LUCET; SCRIPT (Eds.) Nationaler Bildungsbericht Luxemburg 2021 (2021)

Résultats du monitoring scolaire national ÉpStan dans le contexte de la pandémie de COVID-19

in LUCET; SCRIPT (Eds.) Rapport national sur l’éducation au Luxembourg 2021 (2021)

Résultats du monitoring scolaire national ÉpStan dans le contexte de la pandémie de COVID-19 (Matériels supplémentaires)

in LUCET; SCRIPT (Eds.) Rapport National sur l´Éducation au Luxembourg 2021 (2021)

The Impact of the COVID-19 Pandemic on the Luxembourgish Education System: Differences between students based on background characteristics in elementary and secondary school

Scientific Conference (2021, November)

Policy responses to the COVID-19 pandemic (e.g., school closure, home-schooling) have affected students at various stages of education all over the world and were found to increase inequalities in ... [more ▼]

Policy responses to the COVID-19 pandemic (e.g., school closure, home-schooling) have affected students at various stages of education all over the world and were found to increase inequalities in academic achievement (OECD, 2021). The present study is based on fully representative large-scale data from the Luxembourg School Monitoring Programme (Épreuves Standardisées; ÉpStan; LUCET, 2021). The ÉpStan are assessing key competencies of primary and secondary school students in different subjects (e.g., German, French and Math). To allow a fair performance comparison, socio-economic and socio-cultural backgrounds of students (e.g., gender, migration and language background) are systematically taken into consideration. The ÉpStan 2020 entail data from approximatively 25.000 students from five different grades (elementary and secondary school), from 15.000 parents (elementary school) and comparative data from 160.000 students from previous cohorts, thus providing key empirical findings on the pandemic’s impact on the Luxembourgish education system. In the present contribution, we analyze a) how the results of standardized achievement tests compare to previous cohorts and under consideration of students’ socio-economical and socio-cultural background, as well as b) how parents and students perceived home-schooling with regard to aspects such as coping, technical equipment, motivation or contact to teachers. First results indicate that in Grades 1, 5, 7 and 9, standardized achievement scores were generally stable in comparison to previous years. However, in Grade 3, students’ competency scores in German (primary language of instruction in elementary school) listening comprehension worsened substantially. Furthermore, third graders from socio-economically disadvantaged households and/or students that do not speak Luxembourgish/German at home did worse in German reading comprehension than their peers from socio-economically advantaged households and/or speaking Luxembourgish/German at home. Concerning the perception of home-schooling, students coped rather well with the situation, with German being a bit more challenging in primary school and math in secondary school. Findings concerning motivation and enjoyment of home-schooling were mixed, with primary school students’ motivation being comparably to the regular school setting but approximately half of the secondary school students being less motivated than in the regular school setting. Furthermore, all households seem to have been well equipped, with the situation being slightly more favorable in socio-economically advantaged households. For the majority of students, the contact with teachers was frequent, with teachers having adapted their type of support to the needs of their students (e.g., more personal contact towards students from socio-economically disadvantaged households). To conclude, it can be said that no systematic negative trend has been identified in students’ achievement scores. Only German listening comprehension in Grade 3 has worsened substantially and these skills should therefore be fostered as early as possible. Overall, students coped rather well with home-schooling without, however, particularly enjoying it. While students entering the pandemic with favorable background characteristics (e.g., higher socio-economic status, speaking a language of instruction at home) managed better both regarding competencies and perception of home-schooling, students with less favorable background characteristics have received more differentiated support. These findings underline that already existing inequalities in the Luxembourgish school system have in parts been intensified by the pandemic. References LUCET. (2021). Épreuves Standardisées (ÉpStan). https://epstan.lu OECD. (2021). The State of School Education: One Year into the COVID Pandemic. OECD Publishing. https://doi.org/10.1787/201dde84-en [less ▲]