Browse ORBi

- What it is and what it isn't
- Green Road / Gold Road?
- Ready to Publish. Now What?
- How can I support the OA movement?
- Where can I learn more?

ORBi

Are Value-Added Scores Stable Enough for High-Stakes Decisions? Emslander, Valentin ; Levy, Jessica ; et al Scientific Conference (2022, March) Theoretical Background: Can we quantify the effectiveness of a teacher or a school with a single number? Researchers in the field of value-added (VA) models may argue just that (e.g., Chetty et al., 2014 ... [more ▼] Theoretical Background: Can we quantify the effectiveness of a teacher or a school with a single number? Researchers in the field of value-added (VA) models may argue just that (e.g., Chetty et al., 2014; Kane et al., 2013). VA models are widely used for accountability purposes in education and quantify the value a teacher or a school adds to their students’ achievement. For this purpose, these models predict achievement over time and attempt to control for factors that cannot be influenced by schools or teachers (i.e., sociodemographic & sociocultural background). Following this logic, what is left must be due to teacher or school differences (see, e.g., Braun, 2005). To utilize VA models for high-stakes decision-making (e.g., teachers’ tenure, the allocation of funding), these models would need to be highly stable over time. School-level stability over time, however, has hardly been researched at all and the resulting findings are mixed, with some studies indicating high stability of school VA scores over time (Ferrão, 2012; Thomas et al., 2007) and others reporting a lack of stability (e.g., Gorard et al., 2013; Perry, 2016). Furthermore, as there is no consensus on which variables to use as independent or dependent variables in VA models (Everson, 2017; Levy et al., 2019), the stability of VA could vary between different outcome measures (e.g., language or mathematics). If VA models lack stability over time and across outcome measures, their use as the primary information for high-stakes decision-making is in question, and the inferences drawn from them could be compromised. Questions: With these uncertainties in mind, we examine the stability of school VA model scores over time and investigate the differences between language and mathematics achievement as outcome variables. Additionally, we demonstrate the real-life implications of (in)stable VA scores for single schools and point out an alternative, more constructive use of school VA models in educational research. Method: To study the stability of VA scores on school level over time and across outcomes, we drew on a sample of 146 primary schools, using representative longitudinal data from the standardized achievement tests of the Luxembourg School Monitoring Programme (LUCET, 2021). These schools included a heterogeneous and multilingual sample of 7016 students. To determine the stability of VA scores in the subject of mathematics and in languages over time, we based our analysis on two longitudinal datasets (from 2015 to 2017 and from 2017 to 2019, respectively) and generated two VA scores per dataset, one for language and one for mathematics achievement. We further analyzed how many schools displayed stable VA scores in the respective outcomes over two years, and compared the rank correlations of VA scores between language and mathematics achievement as an outcome variable. Results and Their Significance: Only 34-38 % of the schools showed stable VA scores from grade 1 to 3 with moderate rank correlations of r = .37 with language and r = .34 with mathematics achievement. We therefore discourage using VA models as the only information for high-stakes educational decisions. Nonetheless, we argue that VA models could be employed to find genuinely effective teaching or school practices—especially in heterogeneous student populations, such as Luxembourg, in which educational disparities are an important topic already in primary school (Hoffmann et al., 2018). Consequently, we contrast the school climate and instructional quality, which might be a driver of the differences between schools with stable high vs. low VA scores. Literature Braun, H. (2005). Using student progress to evaluate teachers: A primer on value-added models. Educational Testing Service. Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632. https://doi.org/10.1257/aer.104.9.2593 Everson, K. C. (2017). Value-added modeling and educational accountability: Are we answering the real questions? Review of Educational Research, 87(1), 35–70. https://doi.org/10.3102/0034654316637199 Ferrão, M. E. (2012). On the stability of value added indicators. Quality & Quantity, 46(2), 627–637. https://doi.org/10.1007/s11135-010-9417-6 Gorard, S., Hordosy, R., & Siddiqui, N. (2013). How unstable are “school effects” assessed by a value-added technique? International Education Studies, 6(1), 1–9. https://doi.org/10.5539/ies.v6n1p1 Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. Research Paper. MET Project. Bill & Melinda Gates Foundation. https://files.eric.ed.gov/fulltext/ED540959.pdf Levy, J., Brunner, M., Keller, U., & Fischbach, A. (2019). Methodological issues in value-added modeling: An international review from 26 countries. Educational Assessment, Evaluation and Accountability, 31(3), 257–287. https://doi.org/10.1007/s11092-019-09303-w LUCET. (2021). Épreuves Standardisées (ÉpStan). https://epstan.lu Perry, T. (2016). English value-added measures: Examining the limitations of school performance measurement. British Educational Research Journal, 42(6), 1056–1080. https://doi.org/10.1002/berj.3247 Thomas, S., Peng, W. J., & Gray, J. (2007). Modelling patterns of improvement over time: Value added trends in English secondary school performance across ten cohorts. Oxford Review of Education, 33(3), 261–295. https://doi.org/10.1080/03054980701366116 [less ▲] Detailed reference viewed: 12 (2 UL)Stability of Primary School Value-Added Scores over Time: A Comparison Between Math and Language Achievement as Outcome Variables Emslander, Valentin ; Levy, Jessica ; et al Scientific Conference (2021, November) Value-added (VA) models are widely used for accountability purposes in education. Tracking a teacher’s or a school’s VA score over time forms oftentimes the basis for high-stakes decision-making and can ... [more ▼] Value-added (VA) models are widely used for accountability purposes in education. Tracking a teacher’s or a school’s VA score over time forms oftentimes the basis for high-stakes decision-making and can determine whether teachers can keep their jobs or schools may receive certain funding. Despite their high-stakes application, the stability of VA scores over time has not yet been investigated for primary schools. Moreover, it is unclear whether different outcome measures (e.g., language and mathematics) may differ in their stability over time. In the present study, we aimed to clarify the stability of VA scores over time and investigate the differences across outcome variables. Furthermore, we wanted to showcase the real-life implications of (in)stable VA scores for single schools, with a focus on an informative use of VA scores rather than an evaluative way. The exploration of school VA scores in primary schools is especially relevant for heterogeneous student populations, for instance, in Luxembourg. Thus, we drew on representative longitudinal data from the standardized achievement tests of the Luxembourg School Monitoring Programme and examined the stability of school VA scores over two years in 146 schools (N = 7016 students). The overall stability, as measured by correlation coefficients, was moderate with r = .37 for VA scores in language and r = .34 for VA scores in mathematics from grade one to grade three. Real-life implications for schools will be discussed. [less ▲] Detailed reference viewed: 55 (9 UL)Stability of Value-Added Models: Comparing Classical and Machine Learning Approaches Emslander, Valentin ; Levy, Jessica ; et al Scientific Conference (2021, September) Background: What is the value that teachers or schools add to the evolution of students’ performance? Value-added (VA) modeling aims to answer this question by quantifying the effect of pedagogical ... [more ▼] Background: What is the value that teachers or schools add to the evolution of students’ performance? Value-added (VA) modeling aims to answer this question by quantifying the effect of pedagogical actions on students’ achievement, independent of students’ backgrounds (e.g., Braun, 2005). A plethora of VA models exist, and several outcome measures are in use to estimate VA scores, yet without consensus on the model specification (Everson, 2017; Levy et al., 2019). Furthermore, it is unclear whether the most frequently used VA models (i.e., multi-level, linear regression, and random forest models) and outcome measures (i.e., language and mathematics achievement) indicate a similar stability of VA scores over time. Objectives: Drawing from the data of a highly diverse and multilingual school setting, where leveling out the influence of students’ backgrounds is of special interest, we aim to (a) clarify the stability of school VA scores over time; (b) shed light on the sensitivity toward different statistical models and outcome variables; and (c) evaluate the practical implications of (in)stable VA scores for individual schools. Method: Utilizing the representative, longitudinal data from the Luxembourg School Monitoring Programme (LUCET, 2021), we examined the stability of school VA scores. We drew on two longitudinal data sets of students who participated in the standardized achievement tests in Grade 1 in 2014 or 2016 and then again in Grade 3 two years later (i.e., 2016 and 2018, respectively), with a total of 5875 students in 146 schools. School VA scores were calculated using classical approaches (i.e., linear regression and multilevel models) and one of the most commonly used machine learning approaches in educational research (i.e., random forests). Results and Discussion: The overall stability over time across the VA models was moderate, with multilevel models showing greater stability than linear regression models and random forests. Stability differed across outcome measures and was higher for VA models with language achievement as an outcome variable as compared to those with mathematics achievement. Practical implications for schools and teachers will be discussed. [less ▲] Detailed reference viewed: 181 (5 UL)Meta-Analytic Structural Equation Models of Executive Functions and Math Intelligence in Preschool Children Emslander, Valentin ; Scientific Conference (2021, September) BACKGROUND: Response inhibition, attention shifting, and working memory updating are the three core executive functions (EFs; Miyake et al., 2000) underlying other cognitive skills that are relevant for ... [more ▼] BACKGROUND: Response inhibition, attention shifting, and working memory updating are the three core executive functions (EFs; Miyake et al., 2000) underlying other cognitive skills that are relevant for learning and everyday life. For example, they have shown to be differentially related to the mathematical component of intelligence (i.e., math intelligence) in school students and adults. While researchers suppose these three EFs to become more differentiated from early childhood to adulthood, neither the link of these constructs nor their structure has been conclusively established in preschool children yet. Primary studies on path models connecting EFs and math intelligence diverge in the exact relation of EFs and math intelligence. It remains unclear whether inhibition, shifting, and updating exhibit distinct but correlated constructs with respect to their relation to math intelligence. OBJECTIVES: With our meta-analysis, we aimed to (a) synthesize the relation between the three EFs and math intelligence in preschool children; and (b) compare plausible models of the effects of EFs on math intelligence. METHODS/RESULTS: Synthesizing data from 47 studies (363 effect sizes, 30,481 participants) from the last two decades via novel multilevel and multivariate meta-analytic models (Pustejovsky & Tipton, 2020), we found the three core EFs to be significantly related to math intelligence: Inhibition ("r" ̅ = .30, 95 % CI [.25, .35]), shifting ("r" ̅ = .32, 95 % CI [.25, .38]), and updating ("r" ̅ = .36, 95 % CI [.31, .40]). Looking at the three core EFs as one construct, the correlation was "r" ̅ = .34, 95 % CI [.31, .37]. Utilizing correlation-based, meta-analytic structural equation modeling (Jak & Cheung, 2020), our results exhibited significant relations of all EFs to math intelligence. These relations did not differ between the three core EFs. DISCUSSION: Our findings corroborate the positive link between EFs and math intelligence in preschool children and are similar to other age groups. From the model testing, we learned that representing EFs by a latent variable, thus capturing the covariance among the three core EFs, explained substantially more variation in math intelligence than representing them as distinct constructs. [less ▲] Detailed reference viewed: 96 (2 UL)Measuring Executive Functions and their Relation to Math Intelligence in Preschool Children: A Meta-Analysis Emslander, Valentin ; Scientific Conference (2021, July) Introduction: Executive functions (inhibition, attention shifting, updating) are linked to math intelligence in school students and adults. This link is particularly important because performance in ... [more ▼] Introduction: Executive functions (inhibition, attention shifting, updating) are linked to math intelligence in school students and adults. This link is particularly important because performance in school mathematics is predictive of various competencies later in life. While some researchers argue that tests of executive functions and math intelligence measure the same underlying construct, others argue that they measure distinct but correlated constructs. Also, evidence on the differentiation of cognitive skills over time exists. Clarifying the relation between executive functions and math intelligence is, however, challenging, especially because preschoolers cannot fill in commonly used questionnaires that require them to read. As a consequence, researchers have to resort to behavioral, verbal, apparatus-, or computer-based assessments of executive functions. Objectives/Methodology: With this meta-analysis of 29 studies containing 268 effect sizes, we examined the link between executive functions and math intelligence for a total sample of 25,510 preschool children. Specifically, we synthesized the corresponding correlations and sought to clarify which executive function assessments were used for preschool children and how the assessment characteristics may moderate the correlation between executive functions and mathematical skills. Results: Utilizing three-level random-effects meta-analysis, we found a moderate correlation between executive functions and mathematical skills in preschool children, r = 0.35. The type of assessment (behavioral, verbal, apparatus-, or computer-based assessments) did not moderate this relation. Differentiating between the three executive functions revealed average correlations of r = 0.30 between math and inhibition, r = 0.38 between math and attention shifting, and r = 0.36 between math and updating. These analyses will be supplemented by further moderator analyses. Conclusions: Our findings support the significant link between executive functions and mathematical skills in preschoolers—yet, the average correlations do not suggest that both measures are identical. Results will be discussed against the background of deployed assessments and testing environments. [less ▲] Detailed reference viewed: 66 (4 UL)Linking Executive Functions and Math Intelligence in Preschool Children: A Meta-Analysis Emslander, Valentin ; Scientific Conference (2021, May 20) Background: Executive functions (i.e., response inhibition, attention shifting, working memory updating) have shown to be related to the mathematical component of intelligence, which, in turn, is ... [more ▼] Background: Executive functions (i.e., response inhibition, attention shifting, working memory updating) have shown to be related to the mathematical component of intelligence, which, in turn, is predictive of various competences later in life. While this relation has already been thoroughly researched in school students and adults, a comprehensive research synthesis on preschool children—a group for which the assessment of these constructs is more challenging—is still missing. Evidence on the differentiation of cognitive skills over time suggests a differential relation of the three executive functions with math intelligence in older but not in younger children. It remains unclear, however, whether and which one of the three executive functions is more closely related to math intelligence in preschool children. Further research gaps concern the measurement of both executive functions and math intelligence in preschool children, as they cannot complete reading- and writing-based questionnaires. Addressing this measurement challenge, a plethora of inventive measurements has been used to assess both cognitive skills. These measurement differences might also have an influence on the relation between executive functions and math intelligence. Objectives: With our meta-analysis, we aimed to clarify the relation between executive functions and math intelligence in preschool children. Further, we wanted to investigate the influence of different measurement methods on this relation and look into the specific links of inhibition, shifting, and updating with math intelligence more closely. Research questions: 1. To what extent are inhibition, shifting, and updating (as a composite and separately) related to math intelligence in preschool children? (Overall correlations) 2. Which sample, study, and measurement characteristics moderate this relation? (Heterogeneity and moderators) 3. How much variation in math intelligence do inhibition, shifting, and updating explain jointly? (Model testing) Methods: We examined the relation between executive functions and math intelligence for 268 effect sizes from 29 studies for a total sample of 25,510 preschool children. Specifically, we synthesized the corresponding correlations by means of three-level random-effects meta-analyses (RQ 1) and examined the study, sample, and measurement characteristics as possible moderators of this relation between EFs and math intelligence via mixed-effects modeling (RQ 2). Further, we performed meta-analytic structural equation modeling to investigate the joint and differential effects inhibition, shifting, and updating on math intelligence (RQ 3). Results: We found executive functions and math intelligence to correlate moderately in preschool children (r = .35). Investigating inhibition, shifting, and updating separately also revealed moderate average correlations to math intelligence (r = .30, r = .38 , and r = .36, respectively). While we did not find age to explain significant amounts of heterogeneity, four measurement characteristics moderated the relation between executive function and math intelligence. When considered jointly through meta-analytic structural equation modeling, the relations of inhibition, shifting, and updating to math intelligence were similar. Conclusions and Implications: By presenting evidence for a significant relation between executive functions and math intelligence also in preschool children, our findings contribute to the discussion on the differentiation of cognitive skills. They highlight the importance of considering measurement characteristics when researching executive functions and math intelligence. Further, we could not confirm that inhibition, shifting, and updating are differentially related to math intelligence. Further research is needed to clarify the impact of age on the relation between executive functions and math intelligence. [less ▲] Detailed reference viewed: 46 (2 UL)Measuring Executive Functions and their Relations to Mathematical Skills in Preschool Children: A Meta-Analysis Emslander, Valentin ; Scientific Conference (2020, July) Introduction: Executive functions (inhibition, attention shifting, working memory) are linked to mathematical skills in school students and adults. This link is particularly important because performance ... [more ▼] Introduction: Executive functions (inhibition, attention shifting, working memory) are linked to mathematical skills in school students and adults. This link is particularly important because performance in school mathematics is predictive of various competencies later in life. While some researchers argue that tests of executive functions and mathematical skills measure the same underlying construct, others argue that they measure distinct but correlated constructs. Also, evidence on the differentiation of cognitive skills over time exists. Clarifying the relation between executive functions and mathematical skills is, however, challenging, especially because preschoolers cannot fill in commonly used questionnaires that require them to read. As a consequence, researchers have to resort to behavioral, verbal, apparatus-, or computer-based assessments of executive functions. Objectives/Methodology: With this meta-analysis of 26 studies containing 238 effect sizes, we examined the link between executive functions and early mathematical skills for a total sample of 24,256 preschool children. Specifically, we synthesized the corresponding correlations and sought to clarify which executive function assessments were used for preschool children and how the assessment characteristics may moderate the correlation between executive functions and mathematical skills. Results: Utilizing three-level random-effects meta-analysis, we found a moderate correlation between executive functions and mathematical skills in preschool children, r = 0.35. The type of assessment (behavioral, verbal, apparatus-, or computer-based assessments) did not moderate this relation. Differentiating between the three executive functions revealed average correlations of r = 0.31 between math and inhibition, r = 0.38 between math and attention shifting, and r = 0.36 between math and updating. These analyses will be supplemented by further moderator analyses. Conclusions: Our findings support the significant link between executive functions and mathematical skills in preschoolers—yet, the average correlations do not suggest that both measures are identical. Results will be discussed against the background of deployed assessments and testing environments. [less ▲] Detailed reference viewed: 91 (4 UL)Editorial: Advancements in Technology-Based Assessment: Emerging Item Formats, Test Designs, and Data Sources ; ; Greiff, Samuel in Frontiers in Psychology (2020) Detailed reference viewed: 31 (2 UL)Measures of Executive Functions and Mathematical Skills are Distinct Even at a Young Age: A Meta-Analysis with Preschool Children Emslander, Valentin ; Scientific Conference (2020) Measures of executive functions (inhibition, attention shifting, working memory) are linked to measures of mathematical skills in school students and adults. However, the magnitude of this relation in ... [more ▼] Measures of executive functions (inhibition, attention shifting, working memory) are linked to measures of mathematical skills in school students and adults. However, the magnitude of this relation in preschool children is unclear. Following the literature on the differentiation of cognitive skills over time, some researchers suggest that tests of executive functions and mathematical skills measure the same underlying construct, while others suggest that they measure correlated but distinct constructs. This dispute does not only tap the question of how the constructs can be understood but also the question of cost and test efficiency (i.e., assessments of single vs. multiple constructs). Clarifying the relation between measures of the two constructs can be especially challenging because preschoolers cannot fill in commonly used questionnaires that require them to read. Thus, researchers have to resort to behavioral, verbal, apparatus-, or computer-based assessments of executive functions. As a result, executive functions may vary in their relation to mathematical skills as a consequence of their measurement. We examined the link between executive functions and early mathematical skills measures, conducting a meta-analysis of 26 studies containing 238 effect sizes for a total sample of 24,256 preschool children. Specifically, we synthesized the corresponding correlations and aimed to clarify which executive function assessments were used for preschool children and how assessment characteristics may moderate the correlation between executive functions and mathematical skills. Three-level random-effects meta-analysis revealed a small to moderate average correlation between executive functions and mathematical skills measures of preschool children, r = 0.35. The type of assessment (behavioral, verbal, apparatus-, or computer-based assessments) did not moderate this relation. Investigating the three executive functions separately, we found average correlations of r = 0.31 between mathematical skills and inhibition, r = 0.38 between mathematical skills and attention shifting, and r = 0.36 between mathematical skills and updating. These analyses will be supplemented by further moderator and sensitivity analyses. These findings emphasize the significant link between executive functions and mathematical skills measures in preschoolers—hereby, supporting that the measures of both constructs are distinct. In addition, under-researched areas around the assessment of executive functions and mathematical abilities will be discussed. [less ▲] Detailed reference viewed: 93 (5 UL)Complex problem solving and its position in the wider realm of the human intellect Greiff, Samuel ; in Journal of Intelligence (2018) Detailed reference viewed: 213 (8 UL)Critical discussion of the special issue current innovations in computer-based assessment. Greiff, Samuel ; ; in Computers in Human Behavior (2017), 76 Detailed reference viewed: 60 (3 UL)Adaptive problem solving : Moving towards a new assessment domain in the second cycle of PIAAC. Greiff, Samuel ; ; et al in OECD Education Working Papers (2017) Detailed reference viewed: 35 (0 UL)Intelligence in action. Effective strategic behaviors while solving complex problems. ; ; Greiff, Samuel et al in Intelligence (2017), 64 Detailed reference viewed: 155 (7 UL)Current innovations in computer-based assessment. Special issue. ; Greiff, Samuel ; in Computers in Human Behavior (2017) Detailed reference viewed: 72 (4 UL)Editorial to the special issue current innovations in computer-based assessment. ; Greiff, Samuel ; in Computers in Human Behavior (2017), 76 Detailed reference viewed: 38 (1 UL)Crossing the borders of domains. The role of complex problem solving for student learning ; Greiff, Samuel Scientific Conference (2016, November 08) Detailed reference viewed: 151 (6 UL)Using behavioral data from computer-generated log files to understand complex problem solving performance in a computer-based assessment Niepel, Christoph ; Greiff, Samuel ; et al Scientific Conference (2016, July) Detailed reference viewed: 113 (5 UL)Understanding students' performance in a computer-based assessment of complex problem solving. An analysis of behavioral data from computer-generated log files. Greiff, Samuel ; Niepel, Christoph ; et al in Computers in Human Behavior (2016), 61 Computer-based assessments of complex problem solving (CPS) that have been used in international large-scale surveys require students to engage in an in-depth interaction with the problem environment. In ... [more ▼] Computer-based assessments of complex problem solving (CPS) that have been used in international large-scale surveys require students to engage in an in-depth interaction with the problem environment. In this, they evoke manifest sequences of overt behavior that are stored in computer-generated logfiles. In the present study, we explored the relation between several overt behaviors, which N=1476 Finnish ninth-grade students (mean age=15.23,SD=.47 years) exhibited when exploring a CPS environment, and their CPS performance. We used the MicroDYN approach to measure CPS and inspected students' behaviors through log-file analyses. Results indicated that students who occasionally observed the problem environment in a noninterfering way in addition to actively exploring it (noninterfering observation) showed better CPS performance, whereas students who showed a high frequency of (potentially unplanned) interventions (intervention frequency) exhibited worse CPS performance. Additionally, both too much and too little time spent on a CPS task (time on task) was associated with poor CPS performance. The observed effects held after controlling for students' use of an exploration strategy that required a sequence of multiple interventions (VOTAT strategy) indicating that these behaviors exhibited incremental effects on CPS performance beyond the use of VOTAT. [less ▲] Detailed reference viewed: 344 (28 UL)Linking speed and ability in technology-based assessment of complex problem solving ; Greiff, Samuel ; Scientific Conference (2015, April) Detailed reference viewed: 85 (2 UL)Exploring the relation between speed and ability in complex problem solving ; Greiff, Samuel ; in Intelligence (2015), 48 Detailed reference viewed: 145 (7 UL) |
||