Can machine learning methods lead to more precise measures of school effectiveness? An application of various machine learning approaches in the estimation of school value-added scores

Scientific Conference (2020, July)

Contrasting Classical and Machine Learning Approaches in the Estimation of Value-Added Scores in Large-Scale Educational Data

in Frontiers in Psychology (2020), 11

There is no consensus on which statistical model estimates school value-added (VA) most accurately. To date, the two most common statistical models used for the calculation of VA scores are two classical ... [more ▼]

There is no consensus on which statistical model estimates school value-added (VA) most accurately. To date, the two most common statistical models used for the calculation of VA scores are two classical methods: linear regression and multilevel models. These models have the advantage of being relatively transparent and thus understandable for most researchers and practitioners. However, these statistical models are bound to certain assumptions (e.g., linearity) that might limit their prediction accuracy. Machine learning methods, which have yielded spectacular results in numerous fields, may be a valuable alternative to these classical models. Although big data is not new in general, it is relatively new in the realm of social sciences and education. New types of data require new data analytical approaches. Such techniques have already evolved in fields with a long tradition in crunching big data (e.g., gene technology). The objective of the present paper is to competently apply these “imported” techniques to education data, more precisely VA scores, and assess when and how they can extend or replace the classical psychometrics toolbox. The different models include linear and non-linear methods and extend classical models with the most commonly used machine learning methods (i.e., random forest, neural networks, support vector machines, and boosting). We used representative data of 3,026 students in 153 schools who took part in the standardized achievement tests of the Luxembourg School Monitoring Program in grades 1 and 3. Multilevel models outperformed classical linear and polynomial regressions, as well as different machine learning models. However, it could be observed that across all schools, school VA scores from different model types correlated highly. Yet, the percentage of disagreements as compared to multilevel models was not trivial and real-life implications for individual schools may still be dramatic depending on the model type used. Implications of these results and possible ethical concerns regarding the use of machine learning methods for decision-making in education are discussed. [less ▲]

A Formal Framework for Structured N-Back Stimuli Sequences

Scientific Conference (2019, September 15)

A generalizable performance evaluation model of driving games via risk-weighted trajectories

in Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019) (2019)

Efficient learning experiences require content to dynamically match a learner's skill; this assumes a fast and accurate assessment of the learner's skill and the ability to update content accordingly ... [more ▼]

Efficient learning experiences require content to dynamically match a learner's skill; this assumes a fast and accurate assessment of the learner's skill and the ability to update content accordingly. Effective personalized learning therefore involves deriving a performance-predictive mapping between behavioral and environmental factors. Once learned, this relationship can be used to generate new content and to update skill estimates based on the learner's interactions in an adaptive system. To provide proof of concept: (1) We develop a fast-paced driving video game where the player skillfully navigates a cluttered environment comprising obstacles and collectibles. Game content is generated procedurally and player behavior is recorded in the game-this provides an ideal test-bed for a method aiming to learn such a performance-predictive mapping. (2) Using blurred occupancy maps of the game's segments, we generate risk-weighted trajectory profiles for each user and segment of the game. Here, we show that these profiles can be used in a regression model to predict in-game performance both within and between game segments. Additionally, these profiles themselves reveal a trade-off between in-game rewards and risks. Successful identification of predictive environmental units within the game provides insight into the mapping between environmental features and performance, while facilitating the process of procedurally generating new, appropriate content in our adaptive system. We show that rapidly assessed measures of risk are highly predictive of both driving performance and reward rate, providing proof-of-concept evidence for the feasibility of a personalized adaptive learning system for this game. [less ▲]