Spatio-Temporal Representation of Eye-Tracking Data for Cognitive Failure Detection in VR

Human error remains a pervasive risk across safety-critical domains, motivating research on predictive approaches that enable selective automation and adaptive training. This paper presents a longitudinal Virtual Reality (VR) study for cognitive failure prediction, based on 50 hours of eye-tracking data collected over 18 months from a participant solving mental arithmetic problems. We propose a semantic spatio-temporal representation that transforms quantitative eye-tracking features into compact visualizations, and evaluates its effectiveness using machine learning models, including convolutional, recurrent, and fusion neural architectures. These models leverage both spatial and temporal dynamics to detect performance outcomes, achieving up to 87% accuracy in identifying missed responses. However, distinguishing wrong from correct answers proved more difficult, likely due to similar ocular behavior during confident but incorrect responses and the impact of class imbalance. Overall, the results highlight the feasibility of personalized AI systems trained on rich within-subject data, and position VR combined with deep learning as a practical platform for real-time monitoring and adaptive training to reduce human error.