CogPonder: Towards a Computational Framework of General Cognitive Control

Poster (2023, August)

Current computational models of cognitive control exhibit notable limitations. In machine learning, artificial agents are now capable of performing complex tasks but often ignore critical constraints such ... [more ▼]

Current computational models of cognitive control exhibit notable limitations. In machine learning, artificial agents are now capable of performing complex tasks but often ignore critical constraints such as resource limitations and how long it takes for the agent to make decisions and act. Conversely, cognitive control models in psychology are limited in their ability to tackle complex tasks (e.g., play video games) or generalize across a battery of simple cognitive tests. Here we introduce CogPonder, a flexible, differentiable, cognitive control framework that is inspired by the Test-Operate-Test-Exit (TOTE) architecture in psychology and the PonderNet framework in machine learning. CogPonder functionally decouples the act of control from the controlled processes by introducing a controller that acts as a wrapper around any end-to-end deep learning model and decides when to terminate processing and output a response, thus producing both a response and response time. Our experiments show that CogPonder effectively learns from data to generate behavior that closely resembles human responses and response times in two classic cognitive tasks. This work demonstrates the value of this new computational framework and offers promising new research prospects for both psychological and computer sciences. [less ▲]

Validation and Psychometric Analysis of 32 cognitive item models spanning Grades 1 to 7 in the mathematical domain of numbers & operations

Scientific Conference (2022, November)

Today’s educational field has a tremendous hunger for valid and psychometrically sound items to reliably track and model students’ learning processes. Educational large-scale assessments, formative ... [more ▼]

Today’s educational field has a tremendous hunger for valid and psychometrically sound items to reliably track and model students’ learning processes. Educational large-scale assessments, formative classroom assessment, and lately, digital learning platforms require a constant stream of high-quality, and unbiased items. However, traditional development of test items ties up a significant amount of time from subject matter experts, pedagogues and psychometricians and might not be suited anymore to nowadays demands. Salvation is sought in automatic item generation (AIG) which provides the possibility of generating multiple items within a short period of time based on the development of cognitively sound item templates by using algorithms (Gierl & Haladyna, 2013; Gierl et al., 2015). The present study psychometrically analyses 35 cognitive item models that were developed by a team of national subject matter experts and psychometricians and then used for algorithmically producing items for the mathematical domain of numbers & shapes for Grades 1, 3, 5, and 7 of the Luxembourgish school system. Each item model was administered in 6 experimentally varied versions to investigate the impact of a) the context the mathematical problem was presented in, and b) problem characteristics which cognitive psychology identified to influence the problem solving process. Based on samples from Grade 1 (n = 5963), Grade 3 (n = 5527), Grade 5 (n = 5291), and Grade 7 (n = 3018) collected within the annual Épreuves standardisées, this design allows for evaluating whether psychometric characteristics of produced items per model are a) stable, b) can be predicted by problem characteristics, and c) are unbiased towards subgroups of students (known to be disadvantaged in the Luxembourgish school system). After item calibration using the 1-PL model, each cognitive model was analyzed in-depth by descriptive comparisons of resulting IRT parameters, and the estimation of manipulated problem characteristics’ impact on item difficulty by using the linear logistic test model (LLTM, Fischer, 1972). Results are truly promising and show negligible effects of different problem contexts on item difficulty and reasonably stable effects of altered problem characteristics. Thus, the majority of developed cognitive models could be used to generate a huge number of items (> 10.000.000) for the domain of numbers & operations with known psychometric properties without the need for expensive field-trials. We end with discussing lessons learned from item difficulty prediction per model and highlighting differences between the Grades. References: Fischer, G. H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 36, 359-374. Gierl, M. J., & Haladyna, T. M. (Eds.). (2013). Automatic item generation: Theory and practice. New York, NY: Routledge. Gierl, M. J., Lai, H., Hogan, J., & Matovinovic, D. (2015). A Method for Generating Educational Test Items That Are Aligned to the Common Core State Standards. Journal of Applied Testing Technology, 16(1), 1–18. [less ▲]

Linking Theories and Methods in Cognitive Sciences via Joint Embedding of the Scientific Literature: The Example of Cognitive Control

E-print/Working paper (2022)

Traditionally, theory and practice of Cognitive Control are linked via literature reviews by human domain experts. This approach, however, is inadequate to track the ever-growing literature. It may also ... [more ▼]

Traditionally, theory and practice of Cognitive Control are linked via literature reviews by human domain experts. This approach, however, is inadequate to track the ever-growing literature. It may also be biased, and yield redundancies and confusion. Here we present an alternative approach. We performed automated text analyses on a large body of scientific texts to create a joint representation of tasks and constructs. More specifically, 385,705 scientific abstracts were first mapped into an embedding space using a transformers-based language model. Document embeddings were then used to identify a task-construct graph embedding that grounds constructs on tasks and supports nuanced meaning of the constructs by taking advantage of constrained random walks in the graph. This joint task-construct graph embedding, can be queried to generate task batteries targeting specific constructs, may reveal knowledge gaps in the literature, and inspire new tasks and novel hypotheses. [less ▲]

Digitalisation du diagnostic pédagogique : De l’évolution à la révolution

in LUCET; SCRIPT (Eds.) Rapport national sur l’éducation au Luxembourg 2021 (2021)

Digitalisierung der pädagogischen Diagnostik: Von Evolution zu Revolution

in LUCET; SCRIPT (Eds.) Nationaler Bildungsbericht Luxemburg 2021 (2021)

Tackling educational inequalities using school effectiveness measures

Scientific Conference (2020, November 11)

Contrasting classical and machine learning approaches in the estimation of value-added scores in large-scale educational data.

Scientific Conference (2020, March)

Contrasting Classical and Machine Learning Approaches in the Estimation of Value-Added Scores in Large-Scale Educational Data

in Frontiers in Psychology (2020), 11

There is no consensus on which statistical model estimates school value-added (VA) most accurately. To date, the two most common statistical models used for the calculation of VA scores are two classical ... [more ▼]

There is no consensus on which statistical model estimates school value-added (VA) most accurately. To date, the two most common statistical models used for the calculation of VA scores are two classical methods: linear regression and multilevel models. These models have the advantage of being relatively transparent and thus understandable for most researchers and practitioners. However, these statistical models are bound to certain assumptions (e.g., linearity) that might limit their prediction accuracy. Machine learning methods, which have yielded spectacular results in numerous fields, may be a valuable alternative to these classical models. Although big data is not new in general, it is relatively new in the realm of social sciences and education. New types of data require new data analytical approaches. Such techniques have already evolved in fields with a long tradition in crunching big data (e.g., gene technology). The objective of the present paper is to competently apply these “imported” techniques to education data, more precisely VA scores, and assess when and how they can extend or replace the classical psychometrics toolbox. The different models include linear and non-linear methods and extend classical models with the most commonly used machine learning methods (i.e., random forest, neural networks, support vector machines, and boosting). We used representative data of 3,026 students in 153 schools who took part in the standardized achievement tests of the Luxembourg School Monitoring Program in grades 1 and 3. Multilevel models outperformed classical linear and polynomial regressions, as well as different machine learning models. However, it could be observed that across all schools, school VA scores from different model types correlated highly. Yet, the percentage of disagreements as compared to multilevel models was not trivial and real-life implications for individual schools may still be dramatic depending on the model type used. Implications of these results and possible ethical concerns regarding the use of machine learning methods for decision-making in education are discussed. [less ▲]

The Structure of Behavioral Data

E-print/Working paper (2020)

For more than a century, scientists have been collecting behavioral data--an increasing fraction of which is now being publicly shared so other researchers can reuse them to replicate, integrate or extend ... [more ▼]

For more than a century, scientists have been collecting behavioral data--an increasing fraction of which is now being publicly shared so other researchers can reuse them to replicate, integrate or extend past results. Although behavioral data is fundamental to many scientific fields, there is currently no widely adopted standard for formatting, naming, organizing, describing or sharing such data. This lack of standardization is a major bottleneck for scientific progress. Not only does it prevent the effective reuse of data, it also affects how behavioral data in general are processed, as non-standard data calls for custom-made data analysis code and prevents the development of efficient tools. To address this problem, we develop the Behaverse Data Model (BDM), a standard for structuring behavioral data. Here we focus on major concepts in behavioral data, leaving further details and developments to the project's website (https://behaverse.github.io/data-model/). [less ▲]

Principles underlying the design of a cognitive training game as a research framework

in 2019 11th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games) (2019)

Action video games have great potential as cognitive training instruments for their data collection efficiency over standard testing, their natural motive power, and as they have demonstrated benefits for ... [more ▼]

Action video games have great potential as cognitive training instruments for their data collection efficiency over standard testing, their natural motive power, and as they have demonstrated benefits for broad aspects of cognition. However, commercial video games do not allow researchers full control over games' unique features and parameters while presently available scientific games violate key criteria, generally lack appeal, and do not collect enough data for principled exploration of the game design space. To capitalize on the benefits of action video games and facilitate a systematic, scientific exploration of video games and cognition, we propose the Cognitive Training Game Framework (CTGF). The CTGF addresses criteria that we believe are important for gamifying an experimental environment, such as modularity, accessibility, adaptivity, and variety. By offering the potential to collect large data sets and to systematically explore scientific hypotheses in a controlled environment, the resulting framework will make significant contributions to cognitive training research. [less ▲]