Tertium non datur: Various aspects of value-added (VA) models used as measures of educational effectiveness

Value-added (VA) models are used as measures of educational effectiveness which aim to find the “value” that has been added by teachers or schools to students’ achievement, independent of students’ backgrounds. Statistically speaking, teacher or school VA scores are calculated as the part of an outcome variable that cannot be explained by the covariates that are in the VA model (i.e., the residual). Teachers or schools are classified as effective (or ineffective) if they have a positive (or negative) effect on students’ achievement compared to a previously specified norm value. Although VA models have gained popularity in recent years, there is a lack of consensus concerning various aspects of VA scores.
The present dissertation aims at shedding light on these aspects, including the state of the art of VA research in the international literature, covariate choice, and model selection for the estimation of VA scores.
In a first step, a systematic literature review was conducted, in which 370 studies from 26 countries were classified, focusing on methodological issues (Study 1 of the present dissertation). Results indicated no consensus concerning the applied statistical model type (the majority applied a linear regression, followed by multilevel models). Concerning the covariate choice, most studies used prior achievement as a covariate, cognitive and/or motivational student data were hardly considered, and there was no consensus on the in- or exclusion of students’ background variables. Based on these findings, it was suggested that VA models are better suited to improve the quality of teaching than for accountability and decision-making purposes.
Secondly, based on one of the open questions resulting from Study 1 (i.e., covariate choice), the aim of Study 2 was to systematically compare different covariate combinations in the estimation of school VA models. Based on longitudinal data from primary school students participating in the Luxembourg School Monitoring Programme in Grades 1 and 3, three covariate sets were found to be essential when calculating school VA scores with math or language achievement as dependent variables: prior language achievement, prior math achievement, and students’ sociodemographic and sociocultural background. However, the evaluation of individual schools’ effectiveness varied widely depending on the covariate set that was chosen, casting further doubt on the use of VA scores for accountability purposes.
Thirdly, the aim of Study 3 was to investigate statistical model selection, as Study 1 showed no consensus on which model types are most suitable for the estimation of VA scores, with the majority of studies applying linear regression or multilevel models. These classical linear models, along with nonlinear models and different types of machine learning models were systematically compared to each other. Covariates were kept constant (based on the results from Study 2) across models. Multilevel models led to the most accurate prediction of students’ achievement. However, as school VA scores varied depending on specific model choices and as these results can be only generalized for a Luxembourgish sample, it was suggested for future research that the model selection process should be made transparent and should include different specifications in order to obtain ranges of potential VA scores.
In conclusion, all three studies imply that the application of VA models for decision-making and accountability should be critically discussed and that VA scores should not be used as the only measure for accountability or high-stakes decisions. In addition, it can be concluded that VA scores are more suitable for informative purposes. Thus, the findings from the present dissertation prepare the ground for future research, where schools with stable high VA scores can be part of further investigations (both qualitatively and quantitatively) to study their pedagogical strategies and learn from them.