Dispositions for collaborative problem solving.; ;
in Kuger, Susanne; Klieme, Eckhard; Jude, Nina; Kaplan, David (Eds.) Assessing context of learning world-wide (in press)
Dispositions for collaborative problem solving.in Kuger, Susanne; Klieme, Eckhard; Jude, Nina; Kaplan, David (Eds.) Assessing context of learning world-wide (in press)
Religiosity predicts unreasonable coping with COVID-19in Psychology of Religion and Spirituality (in press)
Assessment of problem solvingin Griffin, P.; McGaw, B.; Care, E. (Eds.) Assessment and teaching of 21st century skills (in press)
Well-being and working from home during COVID-19in Information Technology and People (in press)
Purpose – The authors track the well-being of individuals across five European countries during the course of the coronavirus disease 2019 (COVID-19) pandemic and relate their well-being to working from home. The authors also consider the role of pandemic-policy stringency in affecting well-being in Europe. Design/methodology/approach – The authors have four waves of novel harmonised longitudinal data in France, Italy, Germany, Spain and Sweden, covering the period May–November 2020. Well-being is measured in five dimensions: life satisfaction, a worthwhile life, loneliness, depression and anxiety. A retrospective diary indicates whether the individual was working in each month since February 2020 and if so whether at home or not at home. Policy stringency is matched in per country at the daily level. The authors consider both cross- section and panel regressions and the mediating and moderating effects of control variables, including household variables and income. Findings – Well-being among workers is lower for those who work from home, and those who are not working have the lowest well-being of all. The panel results are more mitigated, with switching into working at home yielding a small drop in anxiety. The panel and cross-section difference could reflect adaptation or the selection of certain types of individuals into working at home. Policy stringency is always negatively correlated with well-being. The authors find no mediation effects. The well-being penalty from working at home is larger for the older, the better-educated, those with young children and those with more crowded housing. Originality/value – The harmonised cross-country panel data on individuals’ experiences during COVID-19 are novel. The authors relate working from home and policy stringency to multiple well-being measures. The authors emphasise the effect of working from home on not only the level of well-being but also its distribution.
Facets of conscientiousness and their relation to academic achievement: a person-centered approachScientific Conference (2021, September 16)
Lost in transition – Learning analytics on the transfer from knowledge acquisition to knowledge application in complex problem solvingin Computers in Human Behavior (2021), 115
Since Complex Problem Solving (CPS) skills represent a key competence for educational success, they are of great relevance for learning analytics. More specifically, CPS serves as a pertinent showcase for addressing a crucial existing gap contemporary education is facing, the gap between students' ability to acquire and subsequently apply knowledge in uncertain situations, which are increasingly important in the 21st century. While the CPS process incorporates both the acquisition and application of knowledge, many earlier studies have focused on identifying the factors relevant for success in knowledge acquisition. Given the dearth of existing research on factors influencing a successful transition between both CPS phases, we investigated the rates of successful and unsuccessful knowledge transition over the course of nine CPS items in a sample of N = 1151 students in 9th grade. Results showed that many participants were unable to transition their knowledge from the acquisition to the application phase, which was presumably due to an inefficient mental model transfer. Furthermore, the likelihood of students being ‘lost in transition’ was higher in more complex items. Implications are discussed in light of learning analytics, and particularly with regard to the factors to be taken into account by future CPS training programs.
Students’ Personality Relates to Experienced Variability in State Academic Self-ConceptScientific Conference (2020, November 11)
Attaining a positive academic self-concept (ASC) is linked to many desirable educational outcomes. Research on which student attributes relate to the formation of ASC is therefore considered to be central. Past research on the association between personality traits and ASC has taken an interindividual perspective, while the intraindividual perspective has been disregarded. The present research explored the relation between students’ Big Five traits and intraindividual variability in state general-school ASC in everyday school life for the first time using intensive longitudinal data. We drew on N=294 German ninth and tenth graders who completed a three-week e-diary and a previously presented 60-item Big Five questionnaire (BFI-2; Danner et al., 2016; Soto & John, 2017) assessing Open-Mindedness, Conscientiousness, Extraversion, Agreeableness, and Negative Emotionality as well as their respective subfacets (i.e., resulting in 15 subfacets). To assess state ASC, students completed three items after every single lesson across four different subjects (resulting in Mlessons = 21.12). We ran six mixed-effects location scale models: one specified with all five Big Five domains, and five (one for each Big Five domain) with the subfacets as predictors of intraindividual variability in state ASC. We found that Open-Mindedness, Conscientiousness, Extraversion, and Negative Emotionality as well as at least one subfacet of each Big Five trait were significant predictors of levels of state ASC independently of students’ gender and reasoning ability, and the narrower subfacets Organization (Conscientiousness) and Depression (Negative Emotionality) predicted variability in state ASC independently of students’ gender and reasoning ability. These findings thus provide first evidence that students’ ASC undergoes short-term fluctuations from school lesson to school lesson and that this intraindividual variability can be partly explained by students’ personality. Our results thus contribute to a more complete map of the formation of ASC and the role of personality therein.
The development and validation of a short conscientiousness questionnaire for large-scale educational assessmentScientific Conference (2020, July)
Conscientiousness and its subfacets are related to multiple learning-related outcomes. MacCann, Duckworth and Roberts (2009) developed a questionnaire measuring seven subfacets of conscientiousness with 59 items. However, the resources required to complete such long scales often renders it unsuitable for large-scale educational assessment. Consequently, an economic and psychometrically sound conscientiousness questionnaire that is specifically customized for this context is needed. We developed and validated a short version of the MacCann et al. (2009) questionnaire. In study 1, French and German adaptations of the questionnaire were administered to a representative dataset comprising all ninth-graders in Luxembourg (N1=6325, Cohort 2017). Using an exhaustive search algorithm, we identified the optimal combination of four items for each subfacet by simultaneously considering three criteria: goodness of fit, factor saturation, and scalar measurement invariance across the German and French versions. In study 2, we validated our short 28-item questionnaire on a second, independent sample comprising 6,279 Luxembourgish ninth-graders (Cohort 2018). A 7-factor model assuming separate factors for each subfacet obtained acceptable fit (CFI=.93, RMSEA=.04, SRMR=.06). The criterion validity for each subfacet was tested by examining the relation to standardized achievement tests (SATs). In study 3, drawing on a dataset of 275 tenth-graders (linked longitudinally with the ninth-grade data from study 1), evidence of predictive validity (i.e., school grades) was examined. The subfacets of industriousness, caution and perfectionism showed the strongest relations with both SATs (study 2) and school grades (study 3). Our study delivered a short, valid and reliable questionnaire for the assessment of seven conscientiousness facets in the educational context. The scale is invariant across the German and French language versions and its brevity makes it suitable for large-scale educational assessment.
Self-concept, interest, and achievement within and across math and verbal domains in first- and third-gradersScientific Conference (2020, April)
The generalized internal/external frame-of-reference (G)I/E model explains the formation of domain-specific motivational-affective constructs through social and dimensional comparisons. We examined the associations between verbal and math achievement and corresponding domain-specific academic self-concepts (ASCs) and interests for first-graders and third-graders (N=21,192). Positive achievement-self-concept and achievement-interest relations were found within matching-domains in both grades, while negative cross-domains achievement-self-concept and achievement-interest relations were only found for third-graders. These findings suggest that while the formation of domain-specific ASCs and interests seem to rely on social and dimensional comparisons for third-graders, only social comparisons seem to be in operation for first-graders. Gender and cohort invariance was established in both grade levels. Findings are discussed within the framework of ASC differentiation and dimensional comparison theory.
Are the Machines Taking Over?: Benefits and Challenges of Using Algorithms in (Short) Scale Constructionin European Journal of Psychological Assessment (2020)
Are the machines taking over? Benefits of using algorithms in (short) scale constructionin European Journal of Psychological Assessment (2020), 36
Actually getting some satisfaction. Need-supply fit of fundamental motives at workin Frontiers in Psychology (2020), 11
Actually Getting Some Satisfaction on the Job: Need–Supply Fit of Fundamental Motives at Workin Frontiers in Psychology (2020), 11
Editorial: Advancements in Technology-Based Assessment: Emerging Item Formats, Test Designs, and Data Sourcesin Frontiers in Psychology (2020)
The Multidimensional Forced-Choice Format as an alternative for rating scales. Current state of researchin European Journal of Psychological Assessment (2020), 36
The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans?in Computers in Human Behavior (2020)
The assessment of collaborative problem solving in PISA 2015: An investigation of the validity of the PISA 2015 CPS tasksin Computers and Education (2020)
Generating codebooks to ensure the independent use of research data. Some guidelinesin European Journal of Psychological Assessment (2020), 36
The coronavirus (COVID‐19) fatality risk perception of US adult residents in March and April 2020in British Journal of Health Psychology (2020)
The study compares empirical results on the coronavirus SARS‐CoV‐2 (causing COVID‐19) fatality risk perception of US adult residents stratified for age, gender, and race in mid‐March 2020 (N1 = 1,182) and mid‐April 2020 (N2 = 953). While the fatality risk perception has increased from March 2020 to April 2020, our findings suggest that many US adult residents severely underestimated their absolute and relative fatality risk (i.e., differentiated for subgroups defined by pre‐existing medical conditions and age) at both time points compared to current epidemiological figures. These results are worrying because risk perception, as our study indicates, relates to actual or intended health‐protective behaviour that can reduce SARS‐CoV‐2 transmission rates.
Collaboration in the 21st century: The theory, assessment, and teaching of collaborative problem solvingin Computers in Human Behavior (2020)
Some critical reflections on the special issue: Collaboration in the 21st century: The theory, assessment, and teaching of collaborative problem solvingin Computers in Human Behavior (2020)
Introducing new open science practices at EJPA. European Journal of Psychological Assessmentin European Journal of Psychological Assessment (2020), 36
Using process data to explain group differences in complex problem solvingin Journal of Educational Psychology (2020), 122
Exploring behavioral patterns during complex problem solvingin Journal of Computer Assisted Learning (2020), 36
How working memory capacity and shifting matter for learning. A replication study using worked examplesin Journal of Educational Psychology (2020), 112
Bottom up construction of a personality taxonomyin European Journal of Psychological Assessment (2020), 36
Descriptive, predictive, and explanatory personality research. Different goals, different approaches, but a shared need to move beyond the Big Few traitsin European Journal of Personality (2020), 34
Measuring Anxiety in Older Adults: Development, Diagnostic Accuracy, and Preliminary Validation of a Short-Form of the German Geriatric Anxiety Scale (GAS-G-SF)in Journal of Personality Assessment (2020), 102(2), 196-204
Anxiety symptoms and anxiety disorders are highly prevalent among older adults, and are associ- ated with considerable distress, functional impairment, and burden. Also, there is growing need for brief instruments to measure anxiety symptoms in primary care and geriatric medical settings. Therefore, the current study focuses on the development and psychometric evaluation of a short- form of the Geriatric Anxiety Scale (GAS-G), a well-established anxiety instrument for use with older adults. Study 1 draws on the original data from the GAS-G validation study (N1⁄4242) to develop the short-form (GAS-G-SF) and determines whether the results replicate with the short- form. Study 2 extends the validation of the GAS-G-SF to a clinical sample (N1⁄4156; 62 patients with heart disease, 94 patients with Parkinson’s disease). Overall, the GAS-G-SF showed promising psychometric properties in terms of internal consistency and validity. Also, the GAS-G-SF showed good discriminatory power based on receiver operating characteristic curve analysis in both stud- ies. These results support the utility of the GAS-G-SF as a brief assessment measure for anxiety.
Dimensional and Social Comparison Effects on Domain-Specific Academic Self-Concepts and Interests with First- and Third-Grade StudentsScientific Conference (2019, November 06)
Academic self-concepts (ASCs) are self-perceptions of one’s own academic abilities. The internal/external frame of reference (I/E) model (Marsh, 1986) explains the formation of domain-specific ASCs through a combination of social (i.e. comparing one’s achievement in one domain with the achievement of others in the same domain) and dimensional (i.e. comparing one’s achievement in one domain with one’s achievement in another domain) comparisons. This results into positive achievement-self-concept relations within the math and verbal domains, but into negative achievement-self-concept relations across these domains. The generalized internal/external frame of reference (GI/E) model (Möller, Müller-Kalthoff, Helm, Nagy, & Marsh, 2015) extends the I/E model to the formation of other domain-specific academic self-beliefs such as interest. Research on the validity of the (G)I/E model for elementary school children is limited, especially for first-graders. This study examined the associations between verbal and math achievement and corresponding domain-specific self-concepts and interests for first-graders and third-graders. Two fully representative Luxembourgish first-grader cohorts and two fully representative third-graders cohorts (N=21,192) were used. The analyses were based on structural equation modeling. The findings fully supported the (G)I/E model for third-graders: Achievement was positively related to self-concept and interest within matching domains. Negative relations were found between achievement and self-concept and between achievement and interest across domains. For first-graders, achievement was positively related to self-concept and interest within matching domains. However, the majority of cross-domain relations were non-significant, except for the negative path between math achievement and verbal interest. Hence, while the formation of domain-specific ASCs and interests seem to rely on social and dimensional comparisons for third-graders, only social comparisons seem to be in operation for first-graders. Gender and cohort invariance was established for both grade levels. The findings are discussed within the framework of ASC differentiation and dimensional comparison theory applied to elementary school students.
Examining grades, achievement test scores, and three intelligence facets within an extended I/E modelScientific Conference (2019, September 10)
Domain-specific academic self-concepts (ASCs) are mental representations of one’s abilities that are specific to a particular school subject. According to the internal/external frame of reference (I/E) model, achievement influences the formation of ASCs via social and dimensional comparison processes: within one domain students not only compare their achievement with their peers (social comparison) but also with their own achievements in other domains (dimensional comparison). However, students’ intelligence as an indicator of achievement has been neglected within the I/E model framework. Yet, intelligence is known to be an important determinant of academic success and thus it can be assumed to influence the formation of academic self-concepts, too. Hence, this research examined the links between verbal, numerical, figural intelligence and domain-specific ASCs while controlling for grades and achievement test scores. We drew on N = 382 German students to analyze verbal, numerical, and figural intelligence, German and math achievement, domain-specific self-reported grades and ASCs in math, physics, German, and English. We performed structural equation modeling using Mplus 8 with grades, achievement scores and intelligence facets specified as manifest predictors and domain-specific ASCs as latent criteria. Positive within-domain relations indicating social comparison effects were found between math, physics, German, and English grade to their corresponding self-concept as well as from math achievement to math ASC. Dimensional comparison effects are implied by a positive cross-domain path between physics grade and math ASC and by negative cross-domain paths from math grade to German, physics, and English ASC. Further, a positive cross-domain relation was found between math achievement and physics ASC while a negative cross-domain path was found between German achievement and math ASC. With regard to the intelligence facets, positive paths were found between numerical intelligence and physics ASC as well as between verbal intelligence and English ASC. Our findings thus suggest numerical and verbal intelligence to be valid predictors of ASC formation in an extended I/E model beyond grades and test scores. Overall, the proposed I/E model permitted meaningful relations to be drawn between domain-specific achievement indicators and ASCs suggesting that these variables provide incremental validity.
Entwicklung und Validierung eines Kurzfragebogens zur Erfassung von sieben Facetten von GewissenhaftigkeitScientific Conference (2019, September)
Die Rolle von Persönlichkeitsvariablen für den Schulerfolg rückt immer stärker in den Fokus wissenschaftlicher Untersuchungen. Insbesondere Gewissenhaftigkeit zeigt eine hohe prädiktive Validität für die Schulleistung (Poropat, 2009). Zur näheren Untersuchung des Konstrukts der Gewissenhaft haben MacCann, Duckworth und Roberts (2009) einen aus 68 Items bestehenden Fragebogen zur Erfassung von acht verschiedenen Facetten von Gewissenhaftigkeit im Sekundarschulalter entwickelt. Dieser ist jedoch zu umfangreich für die Verwendung in large-scale Studien, die in der pädagogischen Forschung von zunehmender Bedeutung sind. Der vorliegende Beitrag präsentiert daher die Entwicklung und Validierung einer Kurzform eines Fragebogens zur Erfassung von sieben Facetten von Gewissenhaftigkeit. Die Entwicklungsstichprobe umfasste die Schüler aller neunten Klassen in Luxemburg in 2017 (N1 = 6.325). Die Schüler beantworteten deutsche oder französische Adaptionen eines aus 59 Items und sieben Facetten bestehenden Fragebogens zu Gewissenhaftigkeit, der an das Instrument von MacCann et al. angelehnt war. Zur Entwicklung einer Kurzversion wurde ein exhaustive-search Algorithmus verwendet. Dabei sollte für jede Facette von Gewissenhaftigkeit die bestmögliche Kombination aus vier Items ausgewählt werden. Die Selektionskriterien hierfür waren Fit-Statistiken, interne Konsistenz und Messinvarianz zwischen den Sprachversionen. Der resultierende Fragebogen – bestehend aus 28 Items – wurde 2018 den Schülern aller neunten Klassen in Luxemburg vorgelegt (N2 =6.279). Für diese Validierungsstichprobe zeigte ein Modell mit sieben Faktoren von Gewissenhaftigkeit einen guten Fit (CFI = 0.93, RMSEA = 0.04). Alle Facetten hatten sehr gute Reliabilitäten (ɑs > 0.97). Außerdem fanden wir skalare Messinvarianz zwischen den Sprachversionen und zwischen beiden Geschlechtern. Weitere Validierungsschritte und Anwendungsmöglichkeiten dieses Fragebogens im schulischen Kontext werden diskutiert. Literatur MacCann, C., Duckworth, A.L., & Roberts, R.D. (2009). Empirical identification of the major facets of conscientiousness. Learning and Individual Differences, 19, 451–458. Poropat, A.E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338.
Dimensional Comparison Effects on Facets of Subject-Specific Anxieties: A Nested Modelling ApproachScientific Conference (2019, August 16)
Assimilation and Contrast Effects of Dimensional Comparisons in Self-Concepts, Interests & AnxietiesScientific Conference (2019, August 16)
Research on the internal/external frame of reference (I/E) model has frequently found contrast effects of dimensional comparisons (i.e. a negative relationship between achievement and self-concept across domains) between math and verbal domains. The generalised internal/external frame of reference (GI/E) model extends the I/E model to multiple domains including multiple languages and to other academic self-beliefs and attitudes. When considering multiple languages, achievement-self-concept relations across languages have been found to be either negative (i.e. contrast effect), positive (i.e. assimilation effect), or non-significant. The present study contributes to the ongoing debate concerning the effect of dimensional comparisons among languages by (1) examining dimensional comparisons across two languages and (2) extending the examination to interest and anxiety as outcome variables beyond self-concept. We analysed domain-specific self-concepts, interest, anxieties, and achievement regarding French, German and math in a representative sample (N=5,789) of Luxembourgish ninth-graders. Findings indicated (1) clear contrast effects in the formation of self-concept and interest in German, French and math, and (2) a combination of contrast, assimilation and/or no effects in the formation of anxiety in math, German, and French. With regard to the latter, contrast effects were found for achievement-anxiety paths from German to French, French to German, and French to math. Achievement-anxiety paths from math to French and German to math were non-significant, while the path from math achievement to German anxiety showed a small, yet significant assimilation effect. Results are contextualised within the multilingual Luxembourgish educational system and implications for research on dimensional comparisons are discussed.
Implementing intelligence facets as predictors into the generalised I/E modelScientific Conference (2019, August 13)
The internal/external (I/E) frame of reference model postulates that achievement influences the formation of academic self-concept (ASC) via social and dimensional comparison processes. Its recent extension as the generalized I/E model allows further domains than math and verbal achievement. However, intelligence facets as an indicator of achievement have been neglected within the GI/E model framework. Therefore, the purpose of this research is to explore the influences of three intelligence facets (verbal, numerical, figural) on domain-specific ASCs beyond grades and achievement scores. We drew on N=382 German students to analyze verbal, numerical, and figural intelligence, German and math achievement, self-reported grades and ASCs in four domains. We performed a structural equation model using Mplus with grades, achievement scores and intelligence facets specified as predictors and domain-specific ASCs as criteria. Positive paths were found from math, physics, German and English grade to their corresponding self-concepts. A positive path was found between physics grade and math ASC while negative paths were found from English grade to math and physics ASC and from math grade to German, physics and English ASC. The path coefficients from math achievement to math ASC as well as to physics ASC were positive; German achievement was negatively related to math ASC. Lastly, numerical intelligence was positively related to physics ASC, and verbal intelligence was positively related to English ASC. Thus, intelligence, specifically numerical and verbal intelligence, seem to be valid predictors of domain-specific ACSs beyond grades and achievement test scores.
Thinking in action: Need for Cognition predicts Self-Control together with Action Orientationin PLoS ONE (2019), 14(8),
Developing and validating a short-form assessment of conscientiousness competenciesScientific Conference (2019, August)
Extraversion, working style, reasoning, and complex problem solving. A study on the mechanisms underlying the link between personality and cognitive abilityin Psychological Test and Assessment Modeling (2019), 61
Computer-based collaborative problem solving in PISA 2015 and the role of personalityin Journal of Intelligence (2019), 7
Taking a closer look. An exploratory analysis of successful and unsuccessful strategy use in complex problem solvingin Frontiers in Psychology (2019), 10
The impact of technology on psychological testing in practice and policyin European Journal of Psychological Assessment (2019), 35
Differentiating between static and complex problems: A theoretical framework and its empirical validationin Intelligence (2019), 72
Seeing is believing: Gender diversity in STEM is related to mathematics self-conceptin Journal of Educational Psychology (2019), 111(6), 1119-1130
Mixed Rasch modeling in assessing reading comprehension (Chapter 1)in Quantitative data analysis for language assessment. Volume II: Advanced Methods (2019)
Big-Five personality and political orientation: Results from four panel studies with representative German samplesin Journal of Research in Personality (2019)
The aim of this brief report was to replicate the meta-analytic findings concerning the relationship between Big-Five personality and political orientation reported in Sibley, Osborne, and Duckitt (2012) in a sample of N = 29,015 participants from four panels involving representative German samples. We replicated the expected significant correlations for Openness to Experience (r = ?0.07; 95% CI [?0.10, ?0.05]) and Conscientiousness (r = 0.06, 95% CI [0.05, 0.08]), but the effect sizes were smaller than in Sibley et al. (2012). We also found significant correlations for Agreeableness (r = ?0.04; 95% CI [?0.05, ?0.03]) and Neuroticism (r = ?0.04; 95% CI [?0.06, ?0.02]), indicating small but significant relations of additional Big-Five dimensions on political orientation.
Mixed Rasch modeling in assessing reading comprehensionin Aryadoust, Vahid; Raquel, Michelle (Eds.) Quantitative Data Analysis for Language Assessment (2019)
The English and German versions of the Lesbian, Gay, and Bisexual Identity Scale. Establishing measurement invariance across nationality and gender groupsin Psychology of Sexual Orientation and Gender Diversity (2019), 6
The Incremental Contribution of Complex Problem-Solving Skills to the Prediction of Job Level, Job Complexity, and Salaryin Journal of Business and Psychology (2019), 34
Assessing generic and domain-specific academic competencies in higher educationin Zeitschrift für Pädagogische Psychologie (2019), 33
The nature of faking. A homogeneous and predictable construct?in Psychological Assessment (2019), 31
Why are difficult figural matrices hard to solve? The role of selective encoding and working memory capacityin Intelligence (2019), 72
It is well documented that figural matrices tests are harder to solve when multiple rules need to be induced because multiple rules are traditionally associated with a greater demand for dynamically managed sub-goals (goal management), which requires more working memory capacity (WMC). The current research addresses the necessity to apply selective encoding as a requirement that goes beyond the ability to manage goals when solving figural matrices. In the first study (N = 38), we found that selective encoding demands are present in items with multiple rules in addition to goal management demands. Furthermore, eye movement data indicated that rule induction was hampered when selective encoding demands were present. The second study (N = 127) de-monstrated that individuals' ability to filter relevant features in working memory was positively related to figural matrices items with selective encoding demands. Moreover, there was no evidence that other sources of WMC are related to goal management in figural matrices. Hence, this study provides preliminary evidence that fil-tering of relevant information in working memory is critical for solving figural matrices with multiple rules and challenges the view that goal management is the only driver of the relationship between WMC and performance in solving figural matrices with multiple rules.
The role of planning in complex problem solvingin Computers and Education (2019), 128
Complex problem solving (CPS) is a highly transversal competence needed in educational and vocational settings as well as everyday life. The assessment of CPS is often computer-based, and therefore provides data regarding not only the outcome but also the process of CPS. However, research addressing this issue is scarce. In this article we investigated planning activities in the process of complex problem solving. We operationalized planning through three behavioral measures indicating the duration of the longest planning interval, the delay of the longest planning interval and the variance of intervals between each two successive interactions. We found a significant negative average effect for our delay indicator, indicating that early planning in CPS is more beneficial. However, we also found effects depending on task and interaction effects for all three indicators, suggesting that the effects of different planning behaviors on CPS are highly intertwined.
A need for cognition scale for children and adolescents: Structural analysis and measurement invariancein European Journal of Psychological Assessment (2019), 35
Need for Cognition (NFC) signifies “the tendency for an individual to engage in and enjoy thinking” (Cacioppo & Petty, 1982, p. 116). Up to now, no scale of sufficient psychometric quality existed to assess NFC in children. Using data from three independent, diverse cross-sectional samples from Germany, Luxembourg, and Finland, we examined the psychometric properties of a new NFC scale intended to fill in this gap. In all samples, across grades levels ranging from 1 to 9, confirmatory factor analysis confirmed the hypothesized nested factor structure based on Mussel’s (2013) Intellect model, with one general factor Think influencing all items and two specific factors Seek and Conquer each influencing a subset of items. At least partial scalar measurement invariance with regard to grade level and sex could be demonstrated. The scale exhibited good psychometric properties and showed convergent and discriminant validity with an established NFC scale and other non-cognitive traits such as academic self-concept and interests. It incrementally predicted mostly statistically significant but relatively small portions of academic achievement variance over and above academic self-concept and interest. Implications for research on the development of NFC and its role as an investment trait in intellectual development are discussed.
Complex problem solving. Profiles and developmental paths revealed via latent transition analysisin Developmental Psychology (2019), 55
Replicating and extending the GI/E model: Social and dimensional comparison effects of achievement on test anxiety in math, physics, German, and EnglishScientific Conference (2018, November 09)
The internal/external frame of reference model (I/E model; Marsh, 1986) depicts the formation of academic self-concept by assuming an interplay of social and dimensional comparison processes within and across different domains. Associations between achievement and self-concept are expected to be positive within one given domain due to social comparisons (i.e., external frame of reference) and negative across different domains due to dimensional comparisons (i.e., internal frame of reference). The Generalized I/E model (GI/E model; Möller, Müller-Kalthoff, Helm, Nagy, & Marsh, 2015) allows for the inclusion of other outcome variables besides academic self-concept, for example, interest or motivation. The present research aimed at applying the GI/E model to the construct of test anxiety, thereby replicating and significantly extending some first supportive findings on the validity of the GI/E model with test anxiety (Arens, Becker, & Möller, 2017). To this end, we expanded the scope of the GI/E model with test anxiety for the first time to four domains; namely math, physics, German, and English. For this purpose, we drew on a sample of N = 305 9th and 10th graders attending the highest ability school track (i.e., Gymnasium) from six different secondary schools in four German federal states. Analyses were carried out using structural equation modelling in Mplus 8. We used school grades in the respective domains as achievement indicators. Test anxiety was assessed separately for each domain while simultaneously differentiating between two components of test anxiety; namely, worry and emotionality. Our results suggested negative within-domain relations between achievement and test anxiety in all four domains (i.e., higher grades were associated with less test anxiety). The pattern of results for the relations across domains was not as clear. Positive links between achievement and test anxiety across domains (i.e., higher grades in domain A were associated with higher test anxiety in domain B) were weak and could only be established for the association between German achievement and students’ worry in mathematics. These findings partially supported the assumptions made by the GI/E model. As such, we replicated preexisting findings and added novel insights to the GI/E model. Our results were discussed within the framework of self-concept theory and research as well as dimensional comparison theory.
Developing and validating a short-form questionnaire for the assessment of seven conscientiousness facets in educational large-scale assessmentsScientific Conference (2018, November 09)
The influence of personality on academic outcomes has been demonstrated consistently in previous research. In particular, conscientiousness and its lower order facets generally show the strongest predictive validity for academic success. MacCann et al. (2009) constructed a 68-item questionnaire for the comprehensive assessment of different conscientiousness facets in secondary education. However, such questionnaires are arguably too long for the use in large-scale educational assessments. Investigating the influence of conscientiousness facets within large-scale educational assessments is, nevertheless, crucial for advancing our knowledge of the differential influence of facets on various academic outcomes. Short and psychometrically sound questionnaires are thus required. Currently, no short and yet comprehensive questionnaire assessing the lower order facets of conscientiousness, that is suitable for large-scale educational assessments exists. Therefore, within the present investigation we develop a short-form instrument based upon the seven-factor version (59 items) of the questionnaire presented by MacCann et al. (2009), by using an exhaustive search algorithm and traditional item selection methods. Our sample consists of a large and representative dataset comprising all 9th grade students in Luxembourg from the Luxembourgish national school monitoring system (N = 6325; see epstan.lu), who answered French and German adaptations of the original 59-item questionnaire. We specified the exhaustive search algorithm to select the best possible combination of four items for each lower order facet, by considering goodness of fit criteria, factor saturation statistics, and measurement invariance between the German and French version. In addition, we used Mokken scale analysis and assessed the congruence of item wordings and theoretical definitions of each lower order facet to ensure content validity. We finally selected four to five items per scale based on our analyses. We found good fit statistics for all lower order facet scales (CFI >.95, RMSEA < 0.05), and acceptable to good factor saturation statistics for all but one of these scales (McDonalds ω > 0.7). On top of that, all scales show either scalar invariance, or partial scalar invariance between the German and French language version. The result of our investigation is a short and psychometrically sound instrument for the assessment of the lower order facets of conscientiousness. It is specifically tailored towards the unique context of the multilingual Luxembourgish educational system, and can be used in the upcoming Épstan cohorts.
Dimensional and Social Comparisons Effects on Domain-Specific Self-Concepts and Interests: A Study of Elementary School Children from Luxembourg Across Two WavesScientific Conference (2018, November 09)
Already at the beginning of elementary school, students start to develop self-beliefs and attitudes that reflect their motivation, thoughts, and feelings about a specific school domain. The internal/external frame of reference (I/E) model (Marsh, 1986) encapsulates this phenomenon by explicating the formation of academic self-concept through a combination of social (i.e. comparing one’s achievement in one domain with the achievement of one’s peers in the same domain) and dimensional (i.e. comparing one’s achievement in one domain with one’s achievement in another domain) comparison processes. The recently established generalized internal/external frame of reference (GI/E) model (Möller, Müller-Kalthoff, Helm, Nagy, & Marsh, 2015) builds on the I/E model by extending it to the formation of other subject-specific academic self-beliefs and attitudes, such as interest and test anxiety. Despite the salience of the (G)I/E model in educational research, studies investigating the formation of self-beliefs and attitudes according to this model remains scarce among elementary school children. This study aims to contribute to the current literature by examining the associations between verbal and mathematics achievement, on the one hand, and corresponding domain-specific self-concepts and interests, on the other hand. A population of Luxembourgish elementary school students was assessed twice, two years apart (once in Grade 1 and once in Grade 3) as part of the Luxembourgish school monitoring system (EpStan, cohorts 2014 & 2016; epstan.lu). Using a domain-specific approach, students were assessed on math and verbal achievement as well as German- and math self-concept and interest. This culminated in a sample of N = 3606 elementary school children who participated in both waves. German- and math self-concepts and interests were self-reported whereas standardized achievement tests (see epstan.lu) were used as indicators of math and verbal (Luxembourgish for Grade 1 and German for Grade 3) achievement. Cross-lagged structural equation modelling was performed in Mplus 8 using WLSMV estimation for categorical variables. Overall, the results suggest strong support for the (G)I/E model for Grade 3, while only partial support was found for the (G)I/E model for Grade 1. More specifically, for both Grade 1 and Grade 3, achievement was positively related to self-concept and interest within the same domain. Negative relations between achievement and self-concept and interest across domains were found in Grade 3, but not in Grade 1. The findings are discussed within the context of theory and research on self-concept differentiation processes, dimensional comparison theory, and the multilingual Luxembourgish educational system.
Using Log-File Data to Uncover Strategy Use in Complex Problem SolvingScientific Conference (2018, September 18)
The aim of the current project is to utilize log-file data to enhance the understanding of strategy use in Complex Problem Solving (CPS). CPS can be defined as the ability to manipulate the existing variables of a particular novel, complex, intransparent, and dynamic environment successfully in order to reach a predefined goal. In order to successfully solve a CPS task, the goal-directed application and variation of certain strategies is necessary. To uncover the systematic use of these strategies, log-files have been deemed a fruitful resource as they contain not only the final results of a computer-based CPS item, but also the individual steps undertaken while solving such a task. Recent studies using log-files have highlighted the importance of some strategies for CPS, like the varying one variable at a time (VOTAT) strategy, or of engaging in noninterfering observations (i.e., idle rounds). In addition, other strategies have also been shown to be relevant in neighboring fields of CPS. However, comprehensive studies investigating a broad repertoire of strategies applied in CPS are scarce. Hence, on the basis of existing large-scale assessment data sets, the present project set out to investigate which strategies hidden in log-files are relevant for solving a CPS task in order to gain a more thorough understanding of how CPS tasks are being approached. Preliminary findings indicate that, in addition to VOTAT and idle rounds, particularly the flexibility of switching between strategies should be taken into account.
Problem solving and its role in large-scale assessments. Transversal skills in educational researchScientific Conference (2018, September)
Dynamisches Problemlösen und figurale Matrizenaufgaben. Das MaFIN Item-FormatPoster (2018, September)
Befriedigung gleich Zufriedenheit? Eine Need-Supplies Fit Analyse zur Bedeutung fundamentaler Motive für die ArbeitszufriedenheitScientific Conference (2018, September)
Der Zusammenhang zwischen Facetten komplexer Problemlösefähigkeit, fluider Intelligenz und Arbeitsgedächtniskapazität. Eine umfassende Überprüfung der „different demands“-HypotheseScientific Conference (2018, September)
Die Vorhersage von Selbstkontrolle durch Need for Cognition. Generalisierbarkeit und ErklärungsansätzeScientific Conference (2018, September)
The best of the best. The use of log data to differentiate within ceiling effectsScientific Conference (2018, July)
The role of fundamental motives in job satisfaction. An investigation of needs-supplies fitScientific Conference (2018, July)
Introducing MaFIN. A dynamic matrices finite state automata testScientific Conference (2018, July)
Toward a new model of collaborative problem solving definition and assessmentScientific Conference (2018, July)
An overview. Collaborative problem solving in large-scale assessmentsScientific Conference (2018, April)
Collaborative problem solving human-agent assessment and the role of personality.Scientific Conference (2018, April)
Validating PISA 2015 collaborative problem solving by face-to-face, self-, and teacher report collaboration measures.Scientific Conference (2018, April)
Using process data to explain group differences in complex problem solvingScientific Conference (2018, April)
Interactive effects of comprehension and problem solving skills on digital reading performance and navigationScientific Conference (2018, April)
Wissen ist gut, mehr wissen ist nicht immer besser. Der nonlineare Zusammenhang zwischen komplexem Problemlösen, Intelligenz und VorwissenScientific Conference (2018, February)
Die besten der besten. Der Nutzen von Logdaten zur Differenzierung bei DeckeneffektenScientific Conference (2018, February)
Messinvarianz von Leistungsängstlichkeit über Schulfächer hinweg. Ein Vergleich von Äpfeln mit Birnen?Scientific Conference (2018, February)
Die Erklärung von Geschlechterunterschieden beim komplexen Problemlösen durch ExplorationsverhaltenScientific Conference (2018, February)
„Ich will gut in Mathe sein“. Eine längstschnittliche Überprüfung der Theorie des geplanten Verhaltens zur Vorhersage von MathematikleistungScientific Conference (2018, February)
EJPA Introduces Registered Reports as New Submission Formatin European Journal of Psychological Assessment (2018), 34(4), 217-219
The world beyond rating scales. Why we should think more carefully about the response format in questionnairesin European Journal of Psychological Assessment (2018)
Complex problem solving and its position in the wider realm of the human intellectin Journal of Intelligence (2018)
Collaborative problem solving education for the 21st century workforcein Nature Human Behaviour (2018), 2
Still comparing apples with oranges? Some thoughts on the principles and practices of measurement invariance testingin European Journal of Psychological Assessment (2018), 34
Diagnosing Tony Soprano, Norman Bates, Hercule Poirot, and Carol Beer through the DSM-5 AMPDin European Journal of Psychological Assessment (2018), 34
Would you prefer your coefficients with a little bias, or rather with a lot of variance?in European Journal of Psychological Assessment (2018), 34
Understanding the link between need for cognition and complex problem solvingin Contemporary Educational Psychology (2018), 55
With the aim of broadening the understanding of the nomological network of Complex Problem Solving (CPS), we investigated the link between CPS and Need for Cognition (NFC) while controlling for reasoning ability. Further, we explored whether the relation between NFC and CPS was mediated by test-taking behavior (i.e., exploration time). Using structural equation modeling, we analyzed data from 474 German seventh graders. We found (a) that NFC and CPS were positively related, (b) that this link was partially mediated by CPS exploration time, and (c) that the link between NFC and CPS that was partially mediated by exploration time remained substantial even after we controlled for reasoning. Altogether, these results provide new insights into how CPS and NFC are linked.
Making use of data for assessments. Harnessing analytics and data sciencein Voogt, J.; Knezek, G.; Christensen, R.; Lai, K.-W. (Eds.) International handbook of IT in primary and secondary education (2018)
How employees perceive organizational learning: construct validation of the 25-item short form of the strategic learning assessment map (SF-SLAM)in Journal of Knowledge Management (2018), 22
Students' exploration strategies in computer-simulated complex problem environments: A latent class approachin Computers and Education (2018), 126
Complex problem solving (CPS) is considered an important educational achievement indicator. Previous research has indicated that CPS performance depends to a substantial extent on the way students explore problem environments. In this study, we investigated qualitative differences in the way students interact with such environments. In a sample of N = 2226 Hungarian students in Grades 6 to 8, we applied a latent class approach to investigate the use of the principle of isolated variation as an exploration strategy across six CPS tasks that were developed within the MicroDYN approach. Six qualitatively different class profiles emerged: proficient explorers, intermediate explorers, low-performing explorers, rapid learners, emerging explorers, and nonpersisting explorers. We further validated the profiles by comparing the latent classes with regard to students' overall CPS performance and additional indicators of task exploration. In analyzing age-related and gender differences on a cross-sectional level, we found only a small progression toward better performing class profiles from Grade 6 to Grade 8 (e.g., 14.6% of students in Grade 6 were proficient explorers vs. 24.6% in Grade 8; 27.1% of students in Grade 6 were low-performing explorers vs. 25.8% in Grade 8), and there were no substantial gender differences. This study contributes to the understanding of how students interact with complex problems and is the first to address whether variations in these behaviors indicate qualitatively different levels of strategic behavior. We discuss the theoretical underpinnings and potential of identifying class profiles of students' exploration behavior in the field of educational psychology.
Students' beliefs and attitudes toward mathematics across time: A longitudinal examination of the theory of planned behaviorin Learning and Individual Differences (2018), 63
Advancing the science of collaborative problem solvingin Psychological Science in the Public Interest (2018), 19
Computer-based assessments in PISA. Challenges and potentials for research and policySpeeches/Talks (2017)
Neues zu Prozessindikatoren. Von der Erklärung des Aufgabenerfolgs zum formative AssessmentSpeeches/Talks (2017)
And finally some empirical data: Validating the assessment of Collaborative Problem Solving in PISA.Presentation (2017, October)
Humans vs. Computer-Simulated Agents as Collaboration Partners in Computer-Supported Collaborative Learning EnvironmentsScientific Conference (2017, September)
Dimensional and social comparisons effects on domain-specific self-concepts and interests: An examination of the generalized I/E model in first- and third-gradersScientific Conference (2017, September)
Empirische Trennung von Konnektivität und Dynamik in komplexen ProblemlöseaufgabenScientific Conference (2017, September)
Intrinsische Motive umfassend und ökonomisch messen. Entwicklung und Validierung von 16 Kurzskalen für den Einsatz in der Motivationsforschung.Scientific Conference (2017, September)
Die multidimensionale Erfassung tätigkeitszentrierter Motivdispositionen. Psychometrische Befunde zur Anwendung des LUXXprofiles in Forschung und Praxis.Scientific Conference (2017, September)
Echtzeit- vs. echtzeitnahe Messungen in Paradigmen der differentiell-psychologischen Forschung.Scientific Conference (2017, September)
Organizational learning and complex problem solving. Towards a better understanding of the modern workplace.Scientific Conference (2017, September)
Die Trainierbarkeit von komplexem Problemlösen im Rahmen eines Trainings für Experimentieren.Scientific Conference (2017, September)
The assessment of collaborative problem solving skills – associations with the Big FiveScientific Conference (2017, August)
Collaborative problem solving behavior. A deep dive into log filesScientific Conference (2017, August)
Dimensional comparisons in primary school. A validation of the generalized I/E modelScientific Conference (2017, August)
A broad and multifaceted assessment of intrinsic motivation for research and applied settings: Construct validity of the LUXXprofileScientific Conference (2017, July)
How to measure motivation? Different approaches to the assessment of a complex behaviorScientific Conference (2017, July)
Profiles of students on account of complex problem solving strategiesScientific Conference (2017, July)
To use or not to use? Investigating the validity argument for the use of short scales in research settingsScientific Conference (2017, July)
Intelligence and sensory processing sensitivity. What is the relationship?Scientific Conference (2017, July)
Profiles of students on account of complex problem solving strategies exploited via log-dataScientific Conference (2017, June)
Complex and collaborative problem solving. Transversal skills in educational research.Presentation (2017, May 23)
Explaining increasing performance gaps by test-taking effort in a longitudinal large-scale assessment study in FinlandScientific Conference (2017, April)
Collaborative Problem Solving : Considerations for the National Assessment of Educational ProgressBook published by National Center for Education Statistics (2017)
Effekte sozialer und dimensionaler Vergleiche auf fachspezifische Selbstkonzepte und Interessen bei GrundschulkindernScientific Conference (2017, March)
Mapping an Experiment-Based Assessment of Collaborative Behavior onto Collaborative Problem Solving in PISA 2015 : A Cluster Analysis Approach for Collaborator Profilesin Journal of Educational Measurement (2017), 54(1),
Assimilation and contrast effects in the formation of problem-solving self-conceptin Learning and Individual Differences (2017), 54
The field of psychological assessment. Where it stands and where it’s going. A personal analysis of foci, gaps, and implications for EJPAin European Journal of Psychological Assessment (2017), 33
Mapping an experiment-based collaborative behavior assessment onto collaborative problem solving in PISA 2015. A cluster analysis approach for collaborator profilesin Journal of Educational Measurement (2017), 54
Current methodological issues in educational large-scale assessments. Part 2. Special Issuein Psychological Test and Assessment Modeling (2017)
Psychological Test and Assessment Modeling, Special Issue Current Methodological Issues in Educational Large-Scale Assessments - Part IIin Psychological Test and Assessment Modeling, Special Issue Current Methodological Issues in Educational Large-Scale Assessments – Part II (2017), 59
Editorial to the special issue current methodological issues in educational large-scale assessments. Part 2in Psychological Test and Assessment Modeling (2017), 59
How to make sure your paper is desk rejected. A practical guide to rejection in EJPAin European Journal of Psychological Assessment (2017), 33
Construct validity of a brief multifaceted self-report measure of human motives for research purposesScientific Conference (2017)
How Performing PCA and CFA on the Same Data Equals Troublein European Journal of Psychological Assessment (2017), 33(6), 399-402
A test is much more than just the test. Some thoughts on adaptations and equivalence.in European Journal of Psychological Assessment (2017), 33
The history of complex problem solving (Chapter 7)in Csapo, Beno; Funke, Joachim (Eds.) The nature of problem solving : Using Research to Inspire 21st Century Learning. (2017)
Dynamic problem solving. Multiple-item testing based on minimal complex systems.in Leutner, Detlev; Fleischer, Jens; Grünkorn, Jiliane; Klieme, Eckhard (Eds.) Competence assessment in education : Research, models, and instruments. (2017)
Interactive Problem Solving. Exploring the potential of minimal complex systems (Chapter 6)in Csapo, Beno; Funke, Joachim (Eds.) The nature of problem solving : Using Research to Inspire 21st Century Learning. (2017)
Current innovations in computer-based assessment. Special issue.in Computers in Human Behavior (2017)
Editorial to the special issue current innovations in computer-based assessment.in Computers in Human Behavior (2017), 76
Critical discussion of the special issue current innovations in computer-based assessment.in Computers in Human Behavior (2017), 76
Sometimes more is too much: A rejoinder to the commentaries on Greiff et al. (2015)in Journal of Intelligence (2017), 5
21st century skills at work. The incremental contribution of complex problem solving skills to occupational choice and successScientific Conference (2017)
Intelligenz in Aktion. Eine Logfile-Analyse zum strategischen ProblemlöseverhaltenScientific Conference (2017)
Metacognitive confidence judgments and their link to complex problem solvingin Intelligence (2017)
With the aim to better understand the nature of complex problem solving (CPS), we investigated the link between confidence judgments, which represent a major constituent of metacognitive self-monitoring, and CPS by regressing the two facets of CPS (i.e., knowledge acquisition and knowledge application) on confidence in CPS. To ensure that the link between confidence in CPS and CPS is distinct, we controlled for reasoning, which is the strongest known correlate of CPS. Using structural equation modeling in a sample of 471 German eventh- grade students, we found that confidence in CPS explained 67% of the variance in CPS knowledge acquisition and 55% of the variance in CPS knowledge application. These links were reduced but remained substantial when we controlled for reasoning. The results indicate that confidence judgments as indicators of metacognitive monitoring in CPS are substantially linked to successful CPS, thus bringing us one step closer to a full understanding of CPS.
A primer on relative weights analysis. Illustrations of its utility for management researchersin Psychological Test and Assessment Modeling (2017)
Entwicklung und Validierung eines Messinstrumentes zur Erfassung von systemischem Denken bei Lehramtsstudierenden in ökologischen Kontextenin Zeitschrift für Didaktik der Naturwissenschaften (2017), 23(1), 241--259
Systemisches Denken gilt als zentrale Kompetenz für das Verständnis komplexer Zusammenhänge in der Ökologie, im Nachhaltigkeitsdiskurs und in Kontexten einer Bildung für nachhaltige Entwicklung (BNE). In Studien konnte gezeigt werden, dass systemisches Denken grundsätzlich bei Schülerinnen und Schülern aller Schulstufen sowie bei Erwachsenen gefördert werden kann. Eine wirksame Förderung systemischen Denkens bei Lernenden setzt allerdings voraus, dass die Lehrpersonen selbst systemisch denken können. Daraus erwächst die Forderung, systemisches Denken in die Lehrerausbildung zu integrieren. Für eine Diagnose systemischen Denkens von Lehramtsstudierenden schien jedoch keines der in den bisherigen Studien verwendeten Messinstrumente hinreichend geeignet. Im Rahmen des Projektes SysThema (systems thinking in ecological and multidimensional areas) wurde deshalb ein heuristisches Kompetenzmodell zum systemischen Denken abgeleitet und darauf basierend ein Messinstrument zur Erfassung systemischen Denkens bei Lehramtsstudierenden der Fächer Biologie und Geographie entwickelt. Das Instrument wurde zunächst mehreren Vortestungen unterzogen und dann abschließend an einer Stichprobe von 108 Lehramtsstudierenden erprobt. Es weist bei einem mittleren Schwierigkeitsgrad eine interne Konsistenz (Cronbachs $\alpha$) von 0,75 für den Gesamtwert des systemischen Denkens auf. Zusätzlich wurden Items zum komplexen Problemlösen zur Validierung eingesetzt. Es konnten geringe bis maximal moderate Korrelationen zwischen den Ergebnissen des Tests zum systemischen Denken und des komplexen Problemlösens im Sinne diskriminanter Validität aufgezeigt werden. Das hier vorgestellte Messinstrument kann nun zur Evaluation von Seminaren, in welchen systemisches Denken gefördert werden soll, eingesetzt werden.
Empirical study of computer-based assessment of domain-general complex problem-solving skills (Chapter 8)in Csapo, Beno; Funke, Joachim (Eds.) The nature of problem solving : Using Research to Inspire 21st Century Learning. (2017)
Extension procedures for confirmatory factor analysis.in The Journal of Experimental Education (2017), 85(4), 574-596
Using process data for assessment in Intelligent Tutoring Systems. A psychometrician’s, cognitive psychologist's, and computer scientist’s perspective.in Sottilare, Robert A.; Graesser, Arthur C.; Hu, Xiangen; Goodwin, G. (Eds.) Design recommendations for intelligent tutoring systems. Volume 5 (2017)
Methods for assessing inquiry. Machine-learned and theoretical.in Sottilare, Robert A.; Graesser, Arthur C.; Hu, Xiangen; Goodwin, G. (Eds.) Design recommendations for intelligent tutoring systems. (2017)
Empirical work on computer-based assessment of domain-general complex problem solving skillsin Csapo, Beno; Funke, Joachim (Eds.) The nature of problem solving (2017)
Intelligence and sensory processing sensitivity : What is the relationship?Scientific Conference (2017)
Replikation der Befunde zum OFCI-Modell : Offenheit für Erfahrungen moderiert den Zusammenhang zwischen fluider und kristalliner Intelligenz.Scientific Conference (2017)
Replikation der Befunde zum OFCI Modell. Offenheit für Erfahrungen moderiert den Zusammenhang zwischen fluider und kristaliner IntelligenzScientific Conference (2017)
Die Dauer und die Länge von Fehlnavigationssequenzen beim Lesen digitaler TexteScientific Conference (2017)
Understanding the underpinnings of complex problem solving in a higher-education settingScientific Conference (2017)
Collaborative problem solving in the Programme for International Student Assessment (PISA).in von Davier, Alina A.; Zhu, Mengxiao; Kyllonen, Patrick C. (Eds.) Innovative Assessment of Collaboration (2017)
The Heidelberg inventory of geographic system competency model.in Leutner, Detlev; Fleischer, Jens; Grünkorn, Juliane; Eckhard, Klieme (Eds.) Competence assessment in education : Research, models, and instruments. (2017)
Intelligence in action. Effective strategic behaviors while solving complex problems.in Intelligence (2017), 64
Need for cognition in children and adolescents. Correlates and relations to intelligence and academic performancein Learning and Individual Differences (2017)
Complex Problems in Entrepreneurship Education: Examining Complex Problem-Solving in the Application of Opportunity Identificationin Education Research International (2017)
Adaptive problem solving : Moving towards a new assessment domain in the second cycle of PIAAC.in OECD Education Working Papers (2017)
Validity issues and concerns for technology-based performance assessment.in Sottilare, Robert A.; Graesser, Arthur C.; Hu, Xiangen; Goodwin, G. (Eds.) Design recommendations for intelligent tutoring systems. Volume 5 (2017)
Validity and usefulness of the Highly Sensitive Personality Scale Based on a representative German sampleScientific Conference (2017)
Assessment of collaborative problem solving.in Sottilare, Robert A.; Graesser, Arthur C.; Hu, Xiangen; Goodwin, G. (Eds.) Design recommendations for intelligent tutoring systems. Volume 5 (2017)
Home alone: Complex problem solving performance bene fi ts from individual online assessmentin Computers in Human Behavior (2016)
Computer-based assessments of complex problem solving performance often take place in group settings like classrooms and computer laboratories. Such computer-based procedures provide an excellent opportunity to examine setting effects that might occur while participants are tested in a non-group session online at a time and place of their own choosing. For this purpose, N = 273 teacher students were randomly assigned to one of two settings: the individual online condition (n=216) or the computer laboratory group condition (n=57). Strong factorial measurement invariance was evidenced. Participants performed significantly better in the individual online condition than in the group condition (knowledge acquisition:d=0.38; knowledge application: d=0.39). The worse performance in the group setting compared to the individual setting could neither be explained by exploration time, nor by time on task. The internal experimental design validity strengthens the conclusion that setting-related differences in cognitive ability testing are not negligible but noteworthy.
Crossing the borders of domains. The role of complex problem solving for student learningScientific Conference (2016, November 08)
Results and implications of the PISA 2015 collaborative problem solving validation studySpeeches/Talks (2016)
Computer-based assessment of transversal competence. The case of problem-solving competenceScientific Conference (2016, October)
The role of ICT usage in problem-solving achievement. Findings from the PISA 2012 dataScientific Conference (2016, October)
Computer-based assessment of transversal competence. The case of problem-solving competence.Scientific Conference (2016, October)
Measuring problem solving skills via stealth assessment in an engaging video gamein Computers in Human Behavior (2016), 63
Soziale Vergleiche jenseits der Schule: Gender Diversity in MINT-Berufen sagt das mathematische Selbstkonzept von Schülerinnen und Schülern in 24 Ländern vorausScientific Conference (2016, September 18)
COMPROSO. Ein neues Software-Framework (Open Source) für die webbasierte psychologische DiagnostikScientific Conference (2016, September)
Explaining increasing performance gaps by test-taking effort in a longitudinal large-scale assessment study in FinlandScientific Conference (2016, September)
Assessment of transversal skills in large-scale assessments across countries. What about cross-cultural comparability?in Issues and current topics in cross cultural assessment (2016, July)
Using behavioral data from computer-generated log files to understand complex problem solving performance in a computer-based assessmentScientific Conference (2016, July)
Rethinking web-based testing: COMPROSO, an open source testing tool.Scientific Conference (2016, July)
Lesbian and Gay Identity: Testing for Equivalence Across Germans and US-AmericansScientific Conference (2016, July)
Obtaining behavior-based performance indicators from computer-assisted assessment data. A large-scale approachScientific Conference (2016, May)
The complex route to success: complex problemsolving skills in the prediction of university successin Higher Education Research & Development (2016), 35(2), 365-379
Collaborative problem solving assessment. Challenges and opportunities.Scientific Conference (2016, April)
Educational policy implications of process data obtained in large-scale assessmentsPresentation (2016, April)
Complex problem solving in a changing world of work. An empirical construct validationScientific Conference (2016, April)
The concepts of complex and collaborative problem solving and their assessmentPresentation (2016, March)
Wie entstehen domänenübergreifende Fähigkeiten und Selbstkonzepte? Prädiktoren des ProblemlöseselbstkonzeptsScientific Conference (2016, March)
Welche Rolle spielt metacognitive Selbsteinschätzung beim Lösen komplexer Probleme?Scientific Conference (2016, March)
Problemlösen im übergeordneten Kontext individueller kognitiver FähigkeitenPresentation (2016, February 08)
Prozessindikatoren. Von der Erklärung des Aufgabenerfolgs zum formativen AssessmentScientific Conference (2016, February)
Discussant. In Y. Rosen (Ed.), Collaborative problem solving assessment. Challenges and opportunitiesScientific Conference (2016)
Construct validity of complex problem solving: A comprehensive view on different facets of intelligence and school gradesin Intelligence (2016), 54
Although Complex Problem Solving (CPS) has attracted increasing amounts of attention in recent years (e.g., PISA study), the role of CPS in the nomological network of intelligence is controversial. The question of whether CPS is a distinct construct is as old as CPS research itself, but previous studies have had specific shortcomings when addressing the question of whether CPS is a separable or independent construct. The aim of the present study was, therefore, to combine the advantages of previous studies to facilitate a less biased view of the relation between CPS and established intelligence constructs. A sample of 227 German university students worked on a comprehensive measure of intelligence (Berlin Intelligence Structure test) and two CPS assessment tools (MicroDYN and MicroFIN). Furthermore, final school grades (GPA) served as an external criterion. We applied confirmatory factor analyses and structural equation modeling to investigate the relation between CPS and established intelligence constructs on the basis of different psychometric approaches (i.e., first-order model, nested factor model). Moreover, we examined the incremental validity of CPS in explaining GPA beyond established intelligence constructs. Results indicate that CPS represents unique variance that is not accounted for by established intelligence constructs. The incremental validity of CPS was found only when a commonly used narrow operationalization of intelligence was applied (i.e., figural reasoning) but not when a broad operationalization was applied.
Assessing Collaborative Behavior in Students - An Experiment-Based Assessment Approachin European Journal of Psychological Assessment (2016), 32(1), 52-60
A look back and a glimpse forward. A personal exchange between the current and the incoming editor-in-chief of EJPAin European Journal of Psychological Assessment (2016), 32
Contemporary assessment challenges. The measurement of 21st century skills. Special issuein Applied Measurement in Education (2016)
Easily too difficult. Estimating item difficulty in computer simulated microworldsin Computers in Human Behavior (2016), 65
Guest Editorialin Special Issue: Current Methodological Issues in Educational Large-Scale Assessments – Part I (2016), 58
Editorial to the special issue current methodological issues in educational large-scale assessments. Part 1in Psychological Test and Assessment Modeling (2016), 58
Special Issue: Current Methodological Issues in Educational Large-Scale Assessments – Part Iin Psychological Test and Assessment Modeling (2016), 58
Understanding students' performance in a computer-based assessment of complex problem solving. An analysis of behavioral data from computer-generated log files.in Computers in Human Behavior (2016), 61
Computer-based assessments of complex problem solving (CPS) that have been used in international large-scale surveys require students to engage in an in-depth interaction with the problem environment. In this, they evoke manifest sequences of overt behavior that are stored in computer-generated logfiles. In the present study, we explored the relation between several overt behaviors, which N=1476 Finnish ninth-grade students (mean age=15.23,SD=.47 years) exhibited when exploring a CPS environment, and their CPS performance. We used the MicroDYN approach to measure CPS and inspected students' behaviors through log-file analyses. Results indicated that students who occasionally observed the problem environment in a noninterfering way in addition to actively exploring it (noninterfering observation) showed better CPS performance, whereas students who showed a high frequency of (potentially unplanned) interventions (intervention frequency) exhibited worse CPS performance. Additionally, both too much and too little time spent on a CPS task (time on task) was associated with poor CPS performance. The observed effects held after controlling for students' use of an exploration strategy that required a sequence of multiple interventions (VOTAT strategy) indicating that these behaviors exhibited incremental effects on CPS performance beyond the use of VOTAT.
Complex problem solving skills and innovativeness. Evidence from occupational testing and regional data.in European Journal of Education (2016), 51(2),
The prediction of problem solving assessed via microworlds. The relative importance of fluid reasoning and working memoryin European Journal of Psychological Assessment (2016), 32
Die Rolle transversaler Kompetenzen für schulisches Lernen. Das Beispiel des komplexen Problemlösensin BMBF (Ed.) Forschungsvorhaben in Ankopplung an Large-Scale Assessments (2016)
The choice between what you want now and what you want most. Self-control explains academic achievement beyond cognitive abilityin Personality and Individual Differences (2016), 94
Achieving a university degree is a demanding long-term goal, and students often show varying levels of academic achievement despite similar intellectual abilities. In order to help students, researchers thereby need to understand the origins of these individual differences. However, it remains unclear whether self-control is important for students' academic achievement beyond their general cognitive ability. To answer this question,N= 150 German university students completed a measure of general cognitive ability as well as a German translation of the Brief Self-Control Scale. Grade point average (GPA) served as an objective indicator of academic achievement, complemented by personal ratings as a measure of subjective academic achievement (SAA). Both cognitive ability and self-control explained substantial amounts of variance in GPA; however, only self-control accounted for variance in SAA. The study's keyfinding was that self-control indeed contributed to explaining GPA and SAA, even when cognitive ability was controlled for. On the basis of these results, we argue that self-control holds important explanatory value for both objective and subjective academic achievement, and we discuss the results' practical relevance with regard to student success at university.
Assessment of complex problem solving: What we know and what we don’t knowin Applied Measurement in Education (2016), 29
Complex problem solving in educational contexts. Still something beyond a “good g”?Scientific Conference (2016)
Complex problem solving in educational contexts. Still something beyond a “good g”?in Intelligence (2016), 59
Extension procedures for confirmatory factor analysisin Journal of Experimental Education (2016), 85
Teacher evaluation of student ability. What roles do teacher gender, student gender, and their interaction play?in Hadjar, Andreas; Krolak-Schwerdt, Sabine; Priem, Karin; Glock, S. (Eds.) Gender and educational achievement (2016)
Prediction of Complex Problem Solving and school grades by working memory and ability self-conceptin Learning and Individual Differences (2016), 49
Complex problem solving in a changing world. Bridging domain-specific and transversal competence demands in vocational educationin Mulder, Martin; Winterton, Jonathan (Eds.) Competence-based vocational and professional education. The Springer series Education for the changing world of work (2016)
Individual differences in complex problem solving skills. How they evolve and what they imply.in Journal of Educational Psychology (2016), 108(7), 1028-1044
A short glimpse at the relevance and the measurement of complex problem solving. In Z. Liu (Ed.), Establishing a complex problem solving laboratory in ChinaScientific Conference (2015, December 11)
Transversal skills and human capital. Current directions and future perspectives. In K. Murphy (Ed.), Complex problem solvingScientific Conference (2015, December)
New avenues for educational large-scale assessments. The use of log-file data in PISAScientific Conference (2015, December)
Die Rolle von Geschlecht und Selbstkonzept in der komplexen ProblemlösefähigkeitScientific Conference (2015, September)
ATC21S and OECD PISA. Comparative approaches to the assessment of collaborative problem solving in Germany and AustraliaScientific Conference (2015, September)
Den Zusammenhang zwischen Konfidenz und komplexer Problemlöseleistung verstehen: Eine Betrachtung unter Einbezug von Verhalten, Selbstkonzept und kognitiven Fähigkeiten.Scientific Conference (2015, September)
Need for Cognition bei Kindern und Jugendlichen: Korrelate und Zusammenhänge mit Intelligenz und schulischer LeistungScientific Conference (2015, September)
Exploiting process data. The link of students’ behavior to performance in PISA 2012 problem solving.Scientific Conference (2015, August 28)
Examining the formation of students’ intellectual self-concept and problem-solving self-concept within the revisited internal/external frame of reference modelScientific Conference (2015, August)
Examining students’ mathematics attitudes across time: A test of the theory of planned behavior.Scientific Conference (2015, August)
Challenges of constructing and validating innovative assessments. The example of collaborative problem solving.Scientific Conference (2015, July)
Determinants of individual occupational careers in the 21st century. Does complex problem solving matter beyond general mental ability?Scientific Conference (2015, May 21)
Purpose Complex problem solving (CPS) describes the interaction with dynamic and nonroutine tasks, and has been included in PISA 2012 as a factor for employability. This study examines whether CPS can also contribute to the prediction of career advancement in jobs beyond general mental ability (GMA) as one of the best predictors. Design/Methodology Using latent structural equation modeling (SEM), we analyzed a sample of technicians, service/trade workers, and assemblers (n=245) at a German automotive company. A computer-based assessment measured participants' CPS and GMA levels. The dependent variables were the participants’ job level (ISCO-08) and professional training days. Results CPS and GMA both correlated significantly with career advancement (from .18 to .26, all p < .01). The models showed good fit and indicated that CPS explained incremental variance in one of two indicators (β = .14 for trainings, p < .05; ΔR2 = .02) in comparison with GMA alone (β = .24, p < 0.01; R2= .06). Limitations Analyses did not include processes information from CPS assessment as potential advantage. The company-based sample and cross-sectional data restrict inferences. Research Implications Our findings suggest positive relations between CPS and career advancement even when controlling for GMA. Hence, CPS could be a valuable addition for the study of careers and personnel selection test batteries. Originality/Value The first evaluation of CPS in career research gave a general indication of an as-yet-to-be-defined role of CPS, especially when considering the task characteristics compared to complex and demanding jobs, and process data available through CPS.
Perceiving entrepreneurial challenges as complex problems. The role of complex problem solving in opportunity identification.Scientific Conference (2015, May 21)
Beyond simply correct or incorrect: The psychometric exploitation of behavioral log file data for assessment purposes.Scientific Conference (2015, April 16)
Exploring the potential of computer-generated log files with PISA 2012 problem solving dataScientific Conference (2015, April)
The role of visual-spacial thinking in complex problem solving simulationsPresentation (2015, April)
Complex problem solving within the nomological network of intelligenceScientific Conference (2015, April)
Linking speed and ability in technology-based assessment of complex problem solvingScientific Conference (2015, April)
Domain-general problem solving skills and its role in education in the 21st century.Presentation (2015, April)
The merits of log file analyses. Exemplary findings based on the PISA 2012 problem solving data.Scientific Conference (2015, March 28)
Nutzung und Nutzen von Prozessdaten in computerbasierten TestungenScientific Conference (2015, March 11)
Die ‚Dunkle Seite der Kreativität’: Sagt Kreativität negative Veränderungen im ethischen Entscheiden von Schülerinnen und Schülern vorher?Scientific Conference (2015, March 11)
Log file analyses in educational large-scale assessments: A new window to students’ minds?Presentation (2015, March)
Komplexes Problemlösen und seine Rolle in internationalen Large-Scale Untersuchungen. Ergebnisse eines Projektes zu fachübergreifenden KompetenzenScientific Conference (2015, March)
Komplexes Problemlösen und die Vorhersage von Schulleistung. Präditkiv trotz gutem g?Scientific Conference (2015, March)
Über das Potenzial von Aufgabenbearbeitungszeiten beim komplexen Problemlösen: Vorhersage von schulischen Leistungen und Zusammenhang mit SelbstkonzeptScientific Conference (2015, March)
Adult learning policies in Europe and their effectiveness. Invited panel discussionScientific Conference (2015)
Transversal skills and human capital. Current directions and future perspectives.Scientific Conference (2015)
Assessment of problem solving and higher order thinkingin Spector, J. M. (Ed.) Encyclopedia of Educational Technology (2015)
Die Erfassung naturwissenschaftlicher Kompetenzen bei Luxemburger Schülerinnen und Schülernin Bildungsbericht Luxembourg (2015)
How motivational and cognitive correlates and their interaction influence Complex Problem Solving ProcessesScientific Conference (2015)
The Assessment of 21st Century Skills in Industrial and Organizational Psychology: Complex and Collaborative Problem Solvingin Industrial and Organizational Psychology (2015), 8(2), 1-31
In the current paper, we highlight why and how industrial and organizational psychology can take advantage of research on 21st century skills and their assessment. We present vital theoretical perspectives, a suitable framework for assessment, and exemplary instruments with a focus on advances in the assessment of Human Capital. Specifically, Complex Problem Solving (CPS) and Collaborative Problem Solving (ColPS) are two transversal skills (i.e., skills that span multiple domains) that are generally considered critical in the 21st century workplace. The assessment of these skills in education has linked fundamental research with practical applicability and has provided a useful template for workplace assessment. Both CPS and ColPS capture the interaction of individuals with problems that require the active acquisition and application of knowledge in individual or group settings. To ignite a discussion in industrial and organizational psychology, we discuss advances in the assessment of CPS and ColPS and propose ways to move beyond the current state of the art in assessing job-related skills.
Linkages of complex problem solving and general mental ability to career advancement. Does a transversal skill reveal incremental predictive validity?in International Journal of Lifelong Education (2015), 34
Extending the assessment of Complex Problem Solving to finite state automata. Embracing heterogeneityin European Journal of Psychological Assessment (2015), 31
„Was du heute kannst besorgen…“: Selbstkontrolle erklärt Studienerfolg über Intelligenz hinaus.Scientific Conference (2015)
21st century skills. Recent advancements and international developments. Special Issuein Thinking Skills and Creativity (2015), 8
Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solvingin Computers and Education (2015), 91
More is not always better. The relation between response and response time and their moderation by item and person characteristics in Raven’s matricesin Journal of Intelligence (2015), 3
Sometimes less is more. Comparing the validity of complex problem solving measures.in Intelligence (2015), 50
Exploring the relation between speed and ability in complex problem solvingin Intelligence (2015), 48
Assessing complex problem solving skills with Multiple Complex Systems.in Thinking & Reasoning (2015), 21
Complex problem solving. Facilitating the utilization of a concept towards lifelong education. Special Issuein International Journal of Lifelong Education (2015), 4
The dark side of creativity revisited: Is students’ creativity associated with subsequent decreases in their ethical decision making?in Thinking Skills and Creativity (2015), 18
Multiprofessional collaboration in Finnish schoolsin International Journal of Educational Research (2015), 72
Explaining response confidence in Complex Problem Solving perfomanceScientific Conference (2015)
Die Bedeutung komplexer Problemlösefähigkeit für ein erfolgreiches StudiumScientific Conference (2015)
Assessing analytic and interactive aspects of problem solving competencyin Learning & Individual Differences (2015), 39
Kurzskala zur Messung konstruktivistischer Prozessmerkmale im BiologieunterrichtScientific Conference (2015)
Linking complex problem solving to Innovation via opportunity competencein International Journal of Lifelong Education (2015), 34
Echtzeitbasierte Szenarien zur Erfassung komplexer ProblemlösefähigkeitScientific Conference (2015)
Linking confidence in complex problem solving to problem solving selfconcept, achievement, and need for cognition.Scientific Conference (2015)
Longitudinal evidence for a continuing cognitive cascade. Working memory and fluid intelligence enhance complex problem solving in adolescence.in Frontiers in Psychology (2015), 6
Kurzskala zur Messung konstruktivistischer Prozessmerkmale im Biologieunterrichtin Erkenntnisweg Biologiedidaktik (2015), 14
In biology education, constructivist learning environments are becoming increasingly important. However, there is no time-effective method for measuring characteristics of constructivist processes in biology classes. In this paper, a short questionnaire for the analysis of constructivist process characteristics according to REINMANN and MANDL (2006) based on a longer questionnaire (URHAHNE, MARSCH, WILD & KRÜGER, 2011) will be presented. Items for the short version were selected using content criteria. The instrument covers the features active, self-regulated, emotional, situational, social, and constructive with six subscales and two or three items each. In two studies with 288 students aged 11.9 years and 119 students aged 16.2 years, the measurement reliability and validity is demonstrated by probabilistic and classical test methods. Confirmatory factor analyses show that a sixfactor model is superior to alternative models. Rasch analyses also reveal that the individual scales each represent one facet of a construct and that the scaling on all scales proves to be equidistant. A comparison of the results of the short scale with those of the long version documents adequate construct equivalence. The short questionnaire thus proves to be a reliable and time-effective means for assessing constructivist process characteristics in biology classes.
Collaborative problem solving in large-scale assessments. Experiences from the field.Presentation (2014, December)
Naturwissenschaftliches Denken und Komplexes Problemlösen. Zwei Verwandte.Presentation (2014, October)
Extending the Assessment of Complex Problem Solving to Finite State Automata Embracing HeterogeneityScientific Conference (2014, October)
Problem solving processes in PISA 2012 and the exploitation of computer-generated log files.Presentation (2014, October)
Progress in the LLLight project. Report from the assessment team from Luxembourg.Presentation (2014, October)
Moving towards the assessment of collaborative problem solving skills with a tangible user interfacein Turkish Online Journal of Educational Technology (2014), 13(4), 95-104
The research on the assessment of collaborative problem solving (ColPS), as one crucial 21st Century Skill, is still in its beginnings. Using Tangible User Interfaces (TUI) for this purpose has only been marginally investigated in technology-based assessment. Our first empirical studies focused on light-weight performance measurements, usability, user experience, and gesture analysis to increase our understanding of how people interact with TUI in an assessment context. In this paper we propose a research agenda for assessing ColPS of individuals using the Microworlds methodology implemented on TUIs. In a first example item, we use so-called Micro- DYN items, which are independent microworld scenarios that rely on structural linear equations as underlying model. As the MicroDYN approach has been thoroughly empirically investigated for the assessment of complex problem solving of individuals, it offers a good basis for a reliable and valid assessment. We describe how this approach was applied to create an assessment item for a collaborative setting. This item described in this paper implements a simplified model of a MicroDYN item related to climate change using knowledge of previous studies. Therefore, the focus of the item’s construction lies on meeting the requirements for a standardised high quality assessment. Finally, a research agenda is proposed to sketch the main research issues.
Komplexes Problemlösen und naturwissenschaftliche Erkenntnisgewinnung. Internationale Forschung zu interaktiven Kompetenzen.Presentation (2014, September 25)
Komplexes Problemlösen in Large Scale Assessments: Konstruktvalidität eines innovativen AssessmentsScientific Conference (2014, September 22)
Aktuelle Tests zum Komplexen Problemlösen (KPL) geben an dynamische und interaktive Probleme, wie sie beim Lernen und am Arbeitsplatz immer mehr vorkommen, zu simulieren. Dabei überschneidet sich KPL empirisch teilweise mit allgemeiner Intelligenz. Einzelbefunde sagen zwar Erfolg in Schule und Beruf durch KPL über Intelligenz hinaus vorher, doch spätestens mit der Aufnahme in Large Scale Assessments (LSA) wie PISA steht eine Validierung von KPL auf Konstruktebene aus, um die Anwendbarkeit in LSA empirisch zu fundieren. Ergebnisse zu konvergenter, divergenter und prädiktiver Validität dreier Messinstrumente für KPL sind die Ausganglage für die Fragestellungen, ob (1) drei KPL-Tests empirisch konvergieren, (2) KPL klar abgrenzbar von Intelligenz ist und (3) KPL auf Konstruktebene Schulerfolg vorhersagt? Wir untersuchten eine studentische Stichprobe (n=339) mit KPL-Tests ähnlicher zugrundeliegender Struktur und einem Matrizentest zu schlussfolgerndem Denken als Intelligenzmaß. Schulnoten als Maß für Schulerfolg wurden ebenfalls erhoben. Für Frage (1) kam ein Multitrait-Multimethod-Ansatz zum Einsatz, der die Generalisierbarkeit von Ergebnissen einzelner Messmethoden auf die Konstruktebene empirisch prüft. Für die Fragen (2) und (3), die Zusammenhänge zu schlussfolgerndem Denken und Schulerfolg betrachten, verwendeten wir Strukturgleichungsmodelle. Mit Konsistenzmaßen zwischen .50 und .60 sind die KPL-Tests ausreichend konvergent in dem, was sie messen. Damit sind die KPL-Werte relativ unabhängig von ihren Instrumenten. Wir haben auch Hinweise, dass KPL verschieden ist von Intelligenz, indem letzteres zwei KPL Dimensionen, Wissenserwerb und Wissensanwendung, nur mit βERW=.49 und βANW=.53 vorhersagte. Schulerfolg klärte KPL unabhängig von Intelligenz für naturwissenschaftliche Fächer mit βKPL=.22 auf. Die Ergebnisse bestätigen die Anwendbarkeit von innovativen KPL Assessments auch in sehr großen Erhebungen und stärken nicht zuletzt die Tragfähigkeit von Entscheidungen in der Praxis, die auf solche Ergebnisse zurückgreift und auf deren Validität angewiesen ist.
Komplexes Problemlösen und Intelligenz. Neue Erkenntnisse durch neue Diagnostikinstrumente?Scientific Conference (2014, September)
Collaborative problem solving in PISA 2015. What the national report card can learn from it.Presentation (2014, September)
Complex problem solving. A new curricular perspective for the 21st century.Presentation (2014, September)
Zur Vorhersage des komplexen Problemlösens und der Schulnoten durch das Arbeitsgedächtnis und das FähigkeitsselbstkonzeptScientific Conference (2014, September)
Komplexes Problemlösen und sein Bezug zu Need For CognitionScientific Conference (2014, September)
Predicting teacher judgment and expectations in Finland. A multilevel analysisScientific Conference (2014, September)
Die Vorhersage von Lehrereinschätzungen und Lehrererwartungen in Finnland: Ein mehrebenenanalytischer AnsatzScientific Conference (2014, September)
Determinants of Individual Occupational Careers in the 21st Century – Why Complex Problem Solving Matters Beyond General Mental AbilityScientific Conference (2014, August 27)
The OECD (2010) views problem solving as essential for lifelong learning. Recently, the 2012 PISA cycle administered domain-general problems, which featured the complexity of the structure, dynamics and interconnections of variables, and intransparency of the situation. This emphasis on domain-general problems expresses a paradigmatic shift to lifelong learning. In an attempt towards an integrative process theory of CPS, Fischer, Greiff and Funke (2012) portray knowledge acquisition and knowledge application as the core processes of CPS, which are distinguishable from classical cognitive ability (Dörner, 1986). These core processes are assessed in research via innovative computer-based microworlds. These microworlds simulate complex problems, which share features with real world problems, require knowledge acquisition and knowledge application for a successful task completion and offer a wide range of tasks and approaches (Greiff, Holt, et al., 2013; Greiff, Fischer, et al., 2013). Initiatives looking for the facilitation of so-called 21st century skills (e.g., Griffin, McGaw, & Care, 2012; National Research Council, 2012a; OECD, 2013a, 2013b) strongly relate CPS to demands for professional development as resulting from shifts towards non-routine problem solving and communication tasks in the working lives of adults (e.g., successfully adapting towards IT-depending occupations; Autor, Levy, & Murnane, 2003). So far, the application of CPS in research and practice has mainly been restricted to primary and secondary education. Lifespan or educational research on the role of transversal cognitive skills such as CPS for learning later in life is scarce at best and adult education practice fails to fully tap into the requirements of a working reality (Smith & Reio, 2006), which is defined by uncertainties, contradictions, and dynamic changes. CPS is considered to be important for lifelong learning and occupational success, but underrepresented in research and practice. As a consequence, adult education’s mission to provide opportunities for career preparation (e.g., Gianakos, 1996) and intellectual growth (Perry, 1999) seem to be at risk. Hence, the research goals in this study are to (1) enlarge the understanding for determinants of occupational career indicators (e.g. lifelong learning efforts) and eventually (2) encourage the inclusion of CPS on the adult educational research and practice agenda. Our research goals are built on a rich research fundus of the impact of GMA on professional development (e.g., Schmidt & Hunter, 1998) and promising results on the predictive validity of CPS for academic achievement (e.g., Schweizer, Wüstenberg, & Greiff, 2013; Sonnleitner, Keller, Martin, & Brunner, 2013; Wüstenberg, Greiff, & Funke, 2012) and occupational performance (Abele et al., 2012; Danner et al., 2011). Hence, the targeted constructs are GMA, and CPS as a transversal cognitive skill in occupational careers and professional development.
How gender influences performance assessment. Teacher-student gender interaction in focus.Scientific Conference (2014, August)
Strategies within Complex Problem Solving. Inquiries into exploration behaviour in MicroFINScientific Conference (2014, July)
What companies can gain. The assessment of 21st century skills in personnel selection and human resource developmentPresentation (2014, July)
Need for Cognition in Children and Adolescents: Correlates and Relations to Intelligence and School Performance.Scientific Conference (2014, July)
Computer-Based assessment in the field of education. Benefits and challenges from a psychological perspectiveScientific Conference (2014, May)
Collaborative problem solving. Concept, assessment, and first results.Scientific Conference (2014, April 30)
Political implications of educational large scale assessments on the example of complex problem solving.Scientific Conference (2014, April 30)
Introduction on problem solving and empirical results on the use of strategies in complex problem solving tasks.Scientific Conference (2014, April 30)
The role of strategy knowledge for the application of strategies in complex problem solving tasksin Technology, Knowledge and Learning (2014), 19
Education in the twenty-first century must prepare students to meet the challenges of a dynamic and interconnected world. However, assessment of students’ skills tends to focus primarily on static tasks. Therefore, it is not known whether knowledge about successful strategies displayed on static tasks can be transferred to interactive and dynamic environments. This study investigated whether students’ knowledge of a certain strategy (i.e., vary-one-thing-at-a-time, VOTAT) that was assessed in a paper-and-pencilbased scientific reasoning task as well as their fluid intelligence and learning orientation would be sufficient to explain variance in the application of the VOTAT strategy in solving an interactive complex problem solving (CPS) task (i.e., CPS strategy). Furthermore, we analyzed whether CPS strategy mediated the relation between the predictors (i.e., scientific reasoning, learning orientation, fluid intelligence) and CPS performance. The sample consisted of N = 3,191 Finnish students attending the 6th and 9th grades. Results revealed that all predictors were significantly related to CPS strategy, but a substantial amount of variance in CPS strategy remained unexplained (DeltaR-Square = .583). Furthermore, CPS strategy mediated the relation between the predictors and CPS performance. Three implications are discussed: Different demands on the problem solver, knowledge transfer from static versus interactive tasks, or metastrategic knowledge may explain the unexplained variance in CPS strategy. Additionally, the results of our mediation analyses emphasize the importance of measuring strategies via logfiles to gain a deeper understanding of determinants of students’ CPS performance. Finally, fostering motivational factors such as students’ learning orientation yields small improvements in CPS performance.
Assessment of Complex Problem Solving. What we know and what we don’t knowScientific Conference (2014, April)
Scientific Inquiry. Conception, assessment and first results regarding the assessment tool SINQ (Scientific Inquiry).Scientific Conference (2014, April)
Use of log file data in international large-scale assessments. Analyses on data from PISA 2012Presentation (2014, April)
Validitätsforschung in Large-Scale Assessments. Innovationen ohne Validitätsdilemma am Beispiel des Komplexen ProblemlösensScientific Conference (2014, March)
Exploiting the Potential of Process Data in Complex Problem Solving AssessmentScientific Conference (2014, March)
Die Erfassung komplexer Problemlösefähigkeiten auf Basis Finiter Automaten: Der MicroFIN AnsatzScientific Conference (2014, March)
Konstruktvalidierung von Komplexem Problemlösen mit Hilfe von und für Large-Scale AssessmentsScientific Conference (2014, March)
Die Rolle des Problemlösens in der empirischen Bildungsforschung des 21.JahrhundertsScientific Conference (2014, January)
Cross-national gender differences in complex problem solving and their determinants.in Learning & Individual Differences (2014), 29
The present study examined cross-national gender differences in domain-general complex problem solving (CPS) and their determinants. A CPS test relying on the MicroDYN approach was applied to a sample of 890 Hungarian and German high school students attending 8th to 11th grade. Results based on multi-group confirmatory factor analyses showed that measurement invariance of CPS was found across gender and nationality. Analyses of latent mean differences revealed that males outperformed females and German students outperformed Hungarian students. However, these results were caused by Hungarian females performing worse than all other groups. Further analyses of logfiles capturing process data of the interaction of participants with the task showed that Hungarian females less often used vary-one-thing-at-a-time strategy, which lead to considerably worse knowledge acquisition. Results imply that analyzing process data such as use of strategies is highly advisable to identify determinants of overall performance differences in CPS across groups of interest.
On the relation of Complex Problem Solving, personality, fluid intelligence, and academic achievementin Learning & Individual Differences (2014), 36
What You See Is What You (Don’t) Get: A Comment on Funke’s (2014) Opinion Paperin Frontiers in Psychology (2014), 5
Assessment with microworlds: factor structure, invariance, and latent mean comparison of the MicroDYN testin European Journal of Psychological Assessment (2014), 30
Extending the Assessment of Complex Problem Solving to Finite State Automata: Embracing Heterogeneityin European Journal of Psychological Assessment (2014), Advance Online Publication
Recent advancements in the assessment of Complex Problem Solving (CPS) build on the use of homogenous tasks that enable the reliable estimation of CPS skills. The range of problems featured in established instruments such as MicroDYN is consequently limited to a specific subset of homogeneous complex problems. This restriction is problematic when looking at domain-specific examples of complex problems, which feature characteristics absent from current assessment instruments (e.g., threshold states). We propose to utilize the formal framework of Finite State Automata (FSA) to extend the range of problems included in CPS assessment. An approach based on FSA, called MicroFIN, is presented, translated into specific tasks, and empirically investigated. We conducted an empirical study (N = 576), (1) inspecting the psychometric features of MicroFIN, (2) relating it to MicroDYN, and (3) investigating the relations to a measure of reasoning (i.e., CogAT). MicroFIN (1) exhibited adequate measurement characteristics and multitrait- multimethod models indicated (2) the convergence of latent dimensions measured with MicroDYN. Relations to reasoning (3) were moderate and comparable to the ones previously found for MicroDYN. Empirical results and corresponding explanations are discussed. More importantly, MicroFIN highlights the feasibility of expanding CPS assessment to a larger spectrum of complex problems.
The systematic variation of task characteristics facilitates the understanding of task difficulty: A cognitive diagnostic modeling approach to complex problem solvingin Psychological Test and Assessment Modeling (2014), 56(1), 83-103
Since the 1960ies, when pioneering research on Item Response Theory (IRT) was published, considerable progress has been made with regard to the psychometrical quality of psychological assessment tools. One recent development building upon IRT is the introduction of Cognitive Diagnostic Modeling (CDM). The major goal of introducing CDM was to develop methods that allow for examining which cognitive processes are involved when a person is working on a specific assessment task. More precisely, CDM enables researchers to investigate whether assumed task characteristics drive item difficulty and, thus, person ability parameters. This may – at least according to the assumption inherent in CDM - allow conclusions about cognitive processes involved in assessment tasks. In this study, out of the numerous CDMs available the Least Square Distance Method (LSDM; Dimitrov, 2012) was applied to investigate psychometrical qualities of an assessment instrument measuring Complex Problem Solving (CPS) skills. For the purpose of the study, two task characteristics essential for mastering CPS tasks were identified ex-ante – degree of connectivity and presence of indirect effects by adding eigendynamics to the task. The study examined whether and how the two hypothesized task characteristics drive item difficulty of two CPS dimensions, knowledge acquisition and knowledge application. The sample consisted of 490 German high school students, who completed the computer-based CPS assessment instrument MicroDYN. The two task characteristics in MicroDYN items were varied systematically. Results obtained in LSDM indicated that the two hypothesized task characteristics, degree of connectivity and introducing indirect effects, drove item difficulty only for knowledge acquisition. Hence, other task characteristics that may determine item difficulty of knowledge application need to be investigated in future studies in order to provide a sound measurement of CPS.
Problemlösen in der Pädagogischen Psychologie. Themenheft.in Zeitschrift für Pädagogische Psychologie (2014), 28
Komplexes Problemlösen, schulfachliche Kompetenzen und ihre Relation zu Schulnotenin Zeitschrift für Pädagogische Psychologie (2014), 28(4), 205215
The importance of Complex Problem Solving (CPS) within the educational context is well established. This is one of the reasons why CPS plays a prominent role in educational large-scale assessments (e.g., PISA) besides school competencies. However, recent research on CPS and its connection to school performance did not include such school competencies as such, which have proven to be strong predictors of school grades. Consequently, the aim of this study is to close this gap and to examine the relation between CPS and competencies in mathematics and reading. Based on a sample of N=1908 Finish high school students, structural equation modeling was used to analyse the relation of CPS, school competencies, and school grades. In general, the results showed an incremental predictive power of CPS over and above school competencies on school grades in mathematics and mother language. However, differential effects showed a higher importance of CPS in the mathematic domain in comparison to the language domain especially if controlled for reasoning. Implications for the construct of CPS and its importance within the educational context are discussed.
Editorial zum Themenheft Problemlösen in der Pädagogischen Psychologiein Zeitschrift für Pädagogische Psychologie (2014), 28
Kompetenzmessung im Bildungsbereichin Steffgen, Georges; Michaux, Gilles; Ferring, Dieter (Eds.) Psychologie in Luxemburg - Ein Handbuch (2014)
Computer-based assessment of cross-curricular skills and processes. Special section.in Journal of Educational Psychology (2014)
Editorial to the special section Computer-based assessment of cross-curricular skills and processes.in Journal of Educational Psychology (2014), 106
The development of complex problem solving: a latent growth curve analysisin Journal of Educational and Psychological Consultation (2014), 106
New technologies in psychological assessment. The example of computer-based collaborative problem solving assessment.in International Journal of e-Assessment (2014), 1
Dynamisches Problemlösen stärkt Innovationskompetenzin Rosenberg, B. (Ed.) Strategisches Personalmanagement (2014)
Assessment of collaborative problem solving using linear equations on a tangible tabletopin Kalz, M.; Ras, E. (Eds.) Computer assisted assessment. Research into e-assessment (2014)
Computer-based assessment of Complex Problem Solving in educational contexts and how it is influenced by students’ level of Information and Communication Technology literacyin Journal of Educational Psychology (2014)
Discovering Students’ Complex Problem Solving Strategies in Educational Assessmentin Proceedings of the 7th International Conference on Educational Data Mining. (pp. 225-228) (2014)
Domain-general problem solving skills and education in the 21st centuryin Educational Research Review (2014), 13
Teacher evaluation of student ability: what roles do teacher gender, student gender, and their interaction play?in Educational Research (2014)
Background: Recent decades have been marked by an extensive movement to analyze bias in people’s thinking, especially in gender-related issues. Studies have addressed the question of gender bias in classrooms on different levels—the use of gender in books, learning opportunities determined by students’ gender, or teachers’ gender preferences. Purpose: In this study, we aim to answer the question of whether and under which circumstances the interaction between teacher gender and student gender positively or negatively influences teachers’ evaluations of students’ performance, while controlling for objective measures of students’ performance. For instance, it could be possible that a teacher with the same gender as a student evaluates the student as better than opposite-gender students, independent of their objective performance. Sample: The sample consisted of n > 1,500 Finnish 6th grade students (Mage= 12.67) and their respective class teachers. Design and methods: Students completed several academic skills tests, including a mathematical thinking test, reading comprehension test, and scientific reasoning test. Furthermore, teachers provided their evaluation of each student, evaluating students’ performance in different school subjects and answering questions regarding their probability of academic success. To test whether the teacher-student gender interaction had an effect on the criterion variable, i.e. teachers’ evaluation of the students’ performance, multilevel analyses accounting for between- and within-class effects were applied. Thereby, the effect of students’ objective performance on teachers’ evaluation of the students and main effects of gender were controlled for as covariates. Results: The main results indicated that the interaction between student and teacher gender did not influence teachers’ evaluation of the students. However, regardless of their gender, teachers tended to evaluate girls as better than boys in first language performance (i.e. Finnish language) and potential for success in school. Teacher gender did not influence the evaluation. Conclusions: The results of the study suggest that the interaction between teacher and student gender is unlikely to be a source of possible bias in the evaluations of students in the Finnish educational system.
Digital learning, data rich environments, and computer-based assessment. Special issuein Technology, Knowledge and Learning (2014)
Challenges for education in a connected world. Inaugural to the special issue Digital learning, data rich environments, and computer-based assessmentin Technology, Knowledge and Learning (2014), 19
Domänen-spezifisches und komplexes Problemlösen: Unterschiede und VerknüpfungspunktePresentation (2013, November)
Potentiale technologiebasierter Kompetenzerfassung. Das Beispiel Komplexes ProblemlösenPresentation (2013, October)
Komplexes Problemlösen als eigenständiges KonstruktScientific Conference (2013, September 25)
Recently published studies on Complex Problem Solving (CPS) suggest that assessments of CPS using multiple complex systems are only moderately related to tests of classical cognitive abilities. Further, CPS assessments show incremental validity beyond tests of other cognitive abilities when predicting relevant outcomes. However, these empirical accounts have relied on single CPS assessment instruments. We do not know whether these findings will generalize to the construct level across different CPS assessment instruments. To answer this question, we tested a sample of N = 339 German university students who completed three CPS assessment instruments based on multiple complex systems (MicroDYN, the Genetics Lab, and MicroFIN) and the matrices subtest of the Intelligence Structure Test as measure of reasoning. Students further reported their school grades. Analyses including latent multitrait–multimethod models provided support for the conceptualization of CPS as a complex cognitive ability. Results indicated that different CPS assessment instruments showed sufficient convergent validity (with a consistency mostly between .50 and .60). In addition, we found evidence for the divergent validity of CPS from reasoning (reasoning predicted two CPS facets, knowledge and control, àKNOW = .49 and àCON = .53, respectively). In the prediction of academic achieve- ment, CPS explained variance in natural science grades after we controlled for reasoning (àCPS = .22), whereas social science grades were not predicted. Our findings suggest that the validity of CPS generalizes across different measurement instruments.
Geschlechterbias in der Bildung. Die Interaktion der Geschlechter als UrsacheScientific Conference (2013, September)
Komplexe Problemlösekompetenz. Einfluss von Alter und SchulformScientific Conference (2013, September)
Der MicroFIN-Ansatz. Eine erweiterte Erfassung von ProblemlösefähigkeitScientific Conference (2013, September)
Sagt Problemlösen über Arbeitsgedächtnis und Motivation Schulleistung vorher?Scientific Conference (2013, September)
Classical and transversal cognitive skills in individual’s occupational careersScientific Conference (2013, August)
Relations between problem solving, intelligence, and socio-economic background.Scientific Conference (2013, August)
Determinants of cross-national gender differences in complex problem solving competencyScientific Conference (2013, August)
New technologies in psychological assessment: The example of computer-based collaborative problem solving assessmentScientific Conference (2013, July 09)
Cross-curriculare Kompetenzen in den PISA-Studien: Konzepte, Befunde und offene FragenPresentation (2013, July 04)
Exploring exploration: inquiries into exploration behavior in Complex Problem Solving assessmentin Proceedings of the 6th International Conference on Educational Data Mining (2013, July)
Assessment of Complex Problem Solving: The New MicroFIN ApproachScientific Conference (2013, July)
Classical and transversal skills in individual's occupational careersScientific Conference (2013, May)
Collaborative Problem Solving: current assessment possibilities and issues.Scientific Conference (2013, April 16)
Domain-Specific Problem Solving. Bringing Complex Problem Solving research to school.Scientific Conference (2013, April 16)
Current developments in Europe in the field of technical based assessmentScientific Conference (2013, April 15)
Problem solving: current state of the art and what there is to come.Scientific Conference (2013, April 15)
The development of Complex Problem Solving and its relation to reasoning, age and gender.Scientific Conference (2013, April 15)
News from international large-scale assessments: transversal abilities and their importance in educational assessment.Scientific Conference (2013, April 11)
Beyond Reasoning? Searching for the Construct of Complex Problem SolvingScientific Conference (2013, April)
Complex Problem Solving within the Lifelong Learning ProjectScientific Conference (2013, April)
The potential of Educational Data Mining techniques in educational large-scale assessmentsPresentation (2013, February 28)
Current developments in large-scale assessments and computer-based testingPresentation (2013, February 04)
Transversal skills and their importance in education and educational assessmentPresentation (2013, January 09)
Der Nutzen einer Komplexen Problemlösekompetenz: Theoretische Überlegungen und empirische Befundein Zeitschrift für Pädagogische Psychologie (2013), 27
Seit Jahrzehnten wird diskutiert ob Komplexes Problemlösen (KPL) ausschließlich von Intelligenz und Vorwissen abhängt, oder ob das systematische Generieren von Informationen, die viable Repräsentation der wesentlichen kausalen Zusammenhänge, sowie die zielführende Anwendung von Wissen operative Aspekte von Intelligenz erfordern, die von herkömmlichen Intelligenztests nur unzureichend erfasst werden. Der Klärung dieser Frage stehen bisher messtheoretische Probleme und eine unklare Abgrenzung von KPL zu Vorwissen und Fluider Intelligenz gegenüber. In dieser Arbeit wird daher ein Verfahren zur Messung von KPL-Kompetenz eingeführt, das auf der Verwendung multipler vorwissensneutraler komplexer Probleme basiert. Dieses Verfahren (MicroDYN) wird empirisch an einer studentischen Stichprobe (n = 140) überprüft: Latente Zusammenhänge zu Fluider Intelligenz liegen im Bereich um .50 und bestätigen die empirische Trennbarkeit der Konstrukte. KPL weist inkrementelle Validität gegenüber Fluider Intelligenz bei der Vorhersage von Schulnoten auf, insbesondere für das Gesamtabitur und für naturwissenschaftliche Fächer. Implikationen für den Nutzen von KPL in der Leistungsdiagnostik werden diskutiert.
Measuring Complex Problem Solving: An educational application of psychological theoriesin Journal for Educational Research Online (2013), 5
Complex Problem Solving (CPS) is a central topic in modern educational contexts and has received increased interest in educational large-scale assessment studies such as the Programme for International Student Assessment (PISA) and the Programme for the International Assessment of Adult Competencies (PIAAC). Measurement devices up to the present have suff ered from a lack of theoretical embedment and low reliability. This article reviews the most important theories of CPS that may be applied to the process of rational test construction. Specifi cally, the functionalist approach focusing on cognitive processes and the approach of action theory focusing on distinct phases are discussed in their relation to CPS and its assessment. As an example of how to develop a reliable and valid measurement device based on these theories, we propose the development of MicroDYN, which is the operationalization of CPS in PISA 2012.
Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacityin Learning & Individual Differences (2013), 24
This study examines the validity of the complex problemsolving (CPS) test MicroDYN by investigating a) the relation between its dimensions – rule identification (exploration strategy), rule knowledge (acquired knowledge), rule application (control performance) – and working memory capacity (WMC), and b) whether CPS predicts school grades in different domains beyond WMC. A sample of n=393 German high school students (age M=17.07, SD=1.12) completed the computer-based tests Memory Updating Numerical and the CPS scenario MicroDYN. Using structural equation modeling, WMC predicted rule knowledge and rule application, which remained substantially correlated after controlling forWMC. Rule knowledge predicted school grades in science and social studies beyondWMC, but not in language subjects. Explanations for the differential concurrent validity of CPS as well as prerequisites for valid CPS assessment are discussed.
Perspectives on problem solving in cognitive research and educational assessment: analytical, interactive, and collaborative problem solvingin Journal of Problem Solving (The) (2013), 5
Problem solving has received broad public interest as an important competency in modern societies. In educational large-scale assessments paper-pencil based analytical problem solving was included first (e.g., Programme for International Student Assessment, PISA 2003). With growing interest in more complex situations, the focus has shifted to interactive problem solving (e.g., PISA 2012) requiring identification and control of complex systems. In the future, collaborative problem solving represents the next step in assessing problem solving ability (e.g., PISA 2015). This paper describes these different approaches to assessing problem solving ability in large-scale assessments considering theoretical questions as well as assessment issues. For each of the three types of problem solving, the definition and understanding of the construct is explained, items examples are shown together with some empirical results, and limitations of the respective approach are discussed. A final discussion centers on the connection of cognitive and differential psychology within educational research and assessment.
Inductive reasoning, domain specific and complex problem solving: relations and developmentin Thinking Skills and Creativity (2013), 9
This paper focuses on three different types of reasoning: domain-specific problem solving, complex (general) problem solving, and inductive reasoning. The objective of the study is to examine the differences in the developmental levels of inductive reasoning, domainspecific problem solving, and complex problem solving between three age groups and to describe the relations between the three constructs. The sample was drawn from 3rd to 11th grade students (aged 9–17) in Hungarian primary and secondary schools. There were 300–400 students in each cohort. The internal consistencies of the tests were good: Chronbach ˛ varied between .72 and .95. Each of the skills showed a developmental tendency that could be identified with a logistic curve. In every area the pace of development proved to be relatively slow and the steepest change took place in Grade 7. The bivariate correlations between the three constructs were moderate ranging from .35 to .44 signalling that they do not constitute the same construct. The strength of the relationships between inductive reasoning and complex problem solving proved to be the most stable over time. The correlations between domain-specific and complex problem solving showed an increasing trend over time indicating that the strategies used in different problem solving situations become more similar with age. This study provides evidence that inductive reasoning, domainspecific problem solving and complex problem solving are related but distinct constructs and these skills can be fostered most efficiently between Grades 6 and 8.
A multitrait-multimethod study of assessment instruments for Complex Problem Solvingin Intelligence (2013), 41
Computer-based assessment of Complex Problem Solving: concept, implementation, and applicationin Educational Technology Research and Development (2013), 61
Complex Problem Solving in educational settings – something beyond g: Concept, assessment, measurement invariance, and construct validityin Journal of Educational Psychology (2013), 105
Innovative assessments of cross-curricular competencies such as complex problem solving (CPS) have currently received considerable attention in large-scale educational studies. This study investigated the nature of CPS by applying a state-of-the-art approach to assess CPS in high school. We analyzed whether two processes derived from cognitive psychology, knowledge acquisition and knowledge application, could be measured equally well across grades and how these processes differed between grades. Further, relations between CPS, general mental ability (g), academic achievement, and parental education were explored. Hungarian high school students in Grades 5 to 11 (N 855) completed MicroDYN, which is a computer-based CPS test, and the Culture Fair Test 20-R as a measure of g. Results based on structural equation models showed that empirical modeling of CPS was in line with theories from cognitive psychology such that the two dimensions identified above were found in all grades, and that there was some development of CPS in school, although the Grade 9 students deviated from the general pattern of development. Finally, path analysis showed that CPS was a relevant predictor of academic achievement over and above g. Overall, results of the current study provide support for an understanding of CPS as a cross-curricular skill that is accessible through computer-based assessment and that yields substantial relations to school performance. Thus, the increasing attention CPS has currently received on an international level seems warranted given its high relevance for educational psychologists.
Cross-curricular skills and their assessment in the national Swiss school monitoringScientific Conference (2012, December 07)
How to measure Collaborative Problem Solving? Some item prototypesPresentation (2012, November 12)
Mathematik, Naturwissenschaft und Lesen oder doch noch mehr? Die Rolle fachübergreifender Fähigkeiten in PISAPresentation (2012, October 23)
Connecting PISA 2012 and 2015: Interactive and Collaborative Problem SolvingScientific Conference (2012, October 15)
Technical-based assessment of 21st century skills in an applied contextPresentation (2012, October 15)
Potentials of assessment of 21st century skills in technical environmentsPresentation (2012, October 15)
Problem Solving in educational large scale assessments: A critical look back and a personal outlookScientific Conference (2012, October 12)
Konzepte der biologiedidaktischen Systemdynamik und psychologische Problemlöseszenarien. Ein Integrationsversuch.Presentation (2012, October 08)
Assessment of problem solving as indicator for lifelong learning processes in a business contextScientific Conference (2012, October)
Die Skalierung cross-curricularer Kompetenzen in PISA. Einige Reflexionen am Beispiel des komplexen ProblemlösensScientific Conference (2012, September 25)
Minimal complex systems: A new approach for assessing complex problem solving.Scientific Conference (2012, September 24)
Complex Problem Solving in educational contexts. Assessment, measurement invariance, and construct validityScientific Conference (2012, August 29)
Dimensionality of Analytical and Complex Problem Solving competenceScientific Conference (2012, July)
Prediction of students’ performance on test taking processes in Complex Problem SolvingScientific Conference (2012, July)
Observing teachers’ counseling competence. Evaluating a teacher training program with a behavior observation instrument.Scientific Conference (2012, June 18)
Technical issues and item development for Collaborative Problem Solving in PISA 2015Presentation (2012, May 04)
Large-scale assessments and Collaborative Problem Solving: An impossible mix?Presentation (2012, May 04)
Measuring Complex Problem Solving across class levels: Assessment, invariance, and construct validityScientific Conference (2012, April 27)
Dynamic systems and their importance in competence assessment of problem solvingScientific Conference (2012, April 26)
Potential of data mining and other explorative methods in computer-based assessment: Theoretical considerations and empirical resultsScientific Conference (2012, April 25)
Interactive Problem Solving and its realization in large-scale assessmentsScientific Conference (2012, April 25)
Measuring Complex Problem Solving as a five dimensional constructScientific Conference (2012, April 25)
An authoring tool to create Complex Problem Solving items: The Item BuilderScientific Conference (2012, April 25)
Assessment of complex mental skills in PISA 2015: Unsolved issues and challengesScientific Conference (2012, April 24)
Combining domain specific and domain general problem solving ability: Some ideasScientific Conference (2012, April 24)
Measurement invariance of Complex Problem Solving ability measured by MicroDYNScientific Conference (2012, April 24)
Project introduction: Interactive Problem Solving and lifelong learningScientific Conference (2012, April)
Assessment and theory in Complex Problem Solving. A continuing contradiction?in Journal of Educational and Developmental Psychology (2012), 2
Complex Problem Solving (CPS) describes skills frequently needed in everyday life such as the use of new technological devices. Therefore, CPS skills constitute an increasingly important individual ability that needs theoretically embedded, reliable and validated measurement devices. The present article shows that current tests do not sufficiently address the requirement of a theory-based assessment. An integrative approach, the Action Theoretical Problem Space Model by Rollett (2008), is introduced and used to demonstrate how a theoretical framework can influence and inform test development. Implications for the assessment of CPS and its potential are discussed.
From Interactive to Collaborative Problem Solving: Current issues in the Programme for International Student Assessmentin Review of Psychology = Przegl¸ad psychologiczny : kwartalnik (2012), 19
Item Response Theorie – eine Einführungin Wilde, M.; Basten, S.; Fries, B.; Gröben, C.; Kleindienst-Cachay, I.; Meyer-Ahrens, I. (Eds.) Forschen für den Unterricht. Junge Experten zeigen wie’s geht (2012)
Dynamic Problem Solving: A new measurement perspectivein Applied Psychological Measurement (2012), 36
This article addresses two unsolved measurement issues in dynamic problem solving (DPS) research: (a) unsystematic construction of DPS tests making a comparison of results obtained in different studies difficult and (b) use of time-intensive single tasks leading to severe reliability problems. To solve these issues, the MicroDYN approach is presented, which combines (a) the formal framework of linear structural equation models as a systematic way to construct tasks with (b) multiple and independent tasks to increase reliability. Results indicated that the assumed measurement model that comprised three dimensions, information retrieval, model building, and forecasting, fitted the data well (n = 114 students) and could be replicated in another sample (n = 140), showing excellent reliability estimates for all dimensions. Predictive validity of school grades was excellent for model building but nonexistent for the other two MicroDYN dimensions and for an additional measure of DPS. Implications are discussed.
Complex Problem Solving. More than reasoning?in Intelligence (2012), 40
This study investigates the internal structure and construct validity of Complex Problem Solving (CPS), which is measured by a Multiple-Item-Approach. It is tested, if (a) three facets of CPS – rule identification (adequateness of strategies), rule knowledge (generated knowledge) and rule application (ability to control a system) – can be empirically distinguished, how (b) reasoning is related to these CPS-facets and if (c) CPS shows incremental validity in predicting school grade point average (GPA) beyond reasoning. N=222 university students completed Micro-DYN, a computer-based CPS test and Ravens Advanced Progressive Matrices. Analysis including structural equation models showed that a 2-dimensionsal model of CPS including rule knowledge and rule application fitted the data best. Furthermore, reasoning predicted performance in rule application only indirectly through its influence on rule knowledge indicating that learning during system exploration is a prerequisite for controlling a system successfully. Finally, CPS explained variance in GPA even beyond reasoning, showing incremental validity of CPS. Thus, CPS measures important aspects of academic performance not assessed by reasoning and should be considered when predicting real life criteria such as GPA.
The process of solving complex problemsin Journal of Problem Solving (The) (2012), 4
This article is about Complex Problem Solving (CPS), its history in a variety of research domains (e.g., human problem solving, expertise, decision making, and intelligence), and a formal definition and a process theory of CPS applicable to the interdisciplinary field. CPS is portrayed as (a) knowledge acquisition and (b) knowledge application concerning the goal-oriented control of ystems that contain many highly interrelated elements (i.e., complex systems). The impact of mplicit and explicit knowledge as well as systematic strategy selection on the solution process are discussed, emphasizing the importance of (1) information generation (due to the initial intransparency of the situation), (2) information reduction (due to the overcharging complexity of the problem’s structure), (3) model building (due to the interconnectedness of the variables), (4) dynamic decision making (due to the eigendynamics of the system), and (5) evaluation (due to many, interfering and/or ill-defined goals).
Systemisches und räumliches Denken in der geographischen Bildung. Erste Ergebnisse zur Überprüfung eines Modells der Geographischen Systemkompetenzin Hüttermann, A.; Kirchner, P.; Schuler, S.; Drieling, K. (Eds.) Räumliche Orientierung: Räumliche Orientierung, Karten und Geoinformation im Unterricht (2012)
Fachspezifische Problemlösefähigkeit in gewerblich-technischen Ausbildungsberufen. Modellierung, erreichte Niveaus und relevante Einflussfaktorenin Zeitschrift für Berufs- und Wirtschaftspädagogik (2012), 108
Analytisches und dynamisches Problemlösen im Lichte internationaler Schulleistungsvergleichsstudien: Untersuchungen zur Dimensionalitätin Psychologische Rundschau : Ueberblick Uber die Fortschritte der Psychologie in Deutschland, Oesterreich, und der Schweiz (2012), 63
Der Aufbau von Problemlösekompetenz ist einerseits Ziel schulischen Lernens, andererseits Voraussetzung für erfolgreiches weiteres Lernen. Die PISA-Studie 2003 erbrachte überraschende Befunde: Schülerinnen und Schüler in Deutschland scheinen über kognitives Potenzial zu verfügen, das beim Lösen fächerübergreifender Problemstellungen sichtbar wird, in den Schulen aber nicht hinreichend genutzt wird, um fachliche Kompetenzen aufzubauen. Zur Prüfung dieser Potenzialausschöpfungshypothese bedarf es einer Analyse der kognitiven Anforderungen der PISA-Aufgaben und Studien zur Modellierung der zu ihrer Lösung erforderlichen Kompetenzen bzw. Kompetenzstrukturen. Es werden zwei Studien zum Lösen analytischer und dynamischer Probleme berichtet, deren Ergebnisse für beide Kompetenzbereiche eine dreidimensionale Kompetenzstruktur nahe legen. Substanzielle Korrelationen der einzelnen Dimensionen, insbesondere mit mathematischnaturwissenschaftlichen Kompetenzen, lassen darüber hinaus erwarten, dass gezielte Maßnahmen zur Förderung einzelner Kompetenzdimensionen auf die Entwicklung dieser fachlichen Kompetenzen transferieren.
The Genetics Lab. Acceptance and psychometric characteristics of a computer-based microworld to assess Complex Problem Solvingin Psychological Test and Assessment Modeling (2012), 54
Computer-based problem solving scenarios or “microworlds” are contemporary assessment instruments frequently used to assess students’ complex problem solving behavior – a key aspect of today’s educational curricula and assessment frameworks. Surprisingly, almost nothing is known about their (1) acceptance or (2) psychometric characteristics in student populations. This article introduces the Genetics Lab (GL), a newly developed microworld, and addresses this lack of empirical data in two studies. Findings from Study 1, with a sample of 61 ninth graders, show that acceptance of the GL was high and that the internal consistencies of the scores obtained were satisfactory. In addition, meaningful intercorrelations between the scores supported the instrument’s construct validity. Study 2 drew on data from 79 ninth graders in differing school types. Large to medium correlations with figural and numerical reasoning scores provided evidence for the instrument’s construct validity. In terms of external validity, substantial correlations were found between academic performance and scores on the GL, most of which were higher than those observed between academic performance and the reasoning scales administered. In sum, this research closes an important empirical gap by (1) proving acceptance of the GL and (2) demonstrating satisfactory psychometric properties of its scores in student populations.
Die Bedeutung übergreifender kognitiver Determinanten für die Bewältigung beruflicher Anforderungen. Untersuchung am Beispiel dynamischen und technischen Problemlösensin Zeitschrift für Erziehungswissenschaft (2012), 15
Im Zentrum des Beitrags steht die Bedeutung und Prognosekraft dynamischer Problemlösekompetenz (DPK) für das Kriterium der technischen Problemlöseleistung. Als weitere Prädiktoren wurden in der Untersuchung fluide Intelligenz und technisches Fachwissen berücksichtigt. Die Integration fluider Intelligenz ermöglichte auch eine Untersuchung der Frage, ob sich DKP von fluider Intelligenz unterscheiden lässt. Die auf einer Kfz-Mechatroniker- und Elektronikerstichprobe (n = 129, n = 88) basierenden Analysen belegten die empirische Eigenständigkeit der DKP gegenüber fluider Intelligenz. In der Elektronikerstichprobe erklärte DKP die technische Problemlöseleistung am besten, wobei fluide Intelligenz darüber hinaus inkrementelle Validität zeigte. Bei den Kfz-Mechatronikern korrelierte nur fluide Intelligenz mit dem Kriterium, allerdings schwach. Wurde Fachwissen einbezogen, verschwand sowohl die Bedeutung der DKP als auch fluider Intelligenz. Allerdings übte bei den Elektronikern fluide Intelligenz einen über Wissen vermittelten indirekten Einfluss auf das Kriterium aus. Die nicht ganz eindeutige Befundlage, stichprobenspezifische Unterschiede sowie theoretische Ansätze zur Erklärung der beobachteten Effekte werden diskutiert.
Theoretical extensions: From Interactive to Collaborative Problem SolvingScientific Conference (2011, December 13)
Some thoughts on the assessment of Collaborative Problem Solving.Scientific Conference (2011, December 12)
A framework for assessing individual and collaborative Problem SolvingPresentation (2011, September 28)
Scaling Interactive Problem Solving in German high school students. Results on construct validityScientific Conference (2011, September 26)
Genetics Lab: Intelligenzdiagnostik durch computerbasierte Problemlöseszenarien: Konstruktvalidität, Testmotivation und der Mehrwert von ProzessmaßenScientific Conference (2011, September 26)
Some perspectives on new developments in teaching and educationScientific Conference (2011, September 03)
Dynamic problem solving: Just intelligence or something different?Scientific Conference (2011, September)
Dynamic systems and their importance in competence assessment of problem solvingScientific Conference (2011, September)
Theoretical issues in measuring dynamic problem solving and in defining the domain in PISA 2012Scientific Conference (2011, August 30)
Computer-based assessment and how it affects the construct of problem solvingScientific Conference (2011, July 22)
Dynamic problem solving competency: More than intelligence?Scientific Conference (2011, July 05)
Assessment of dynamic problem solving competency with computer-based problemsScientific Conference (2011, July 04)
The Genetics Lab: On the usability and psychometric properties of a newly developed microworld to assess Complex Problem Solving in student samplesScientific Conference (2011, April 27)
Measuring a cross-curricular skill: competence models and construct validity of dynamic problem solvingScientific Conference (2011, April 07)
Systemisches und räumliches Denken in der geographischen Bildung. Erste Ergebnisse zur Überprüfung eines Modells der Geographischen SystemkompetenzScientific Conference (2011, April 06)
Dynamic Problem Solving: A new computer-based perspective for large-scale assessmentsScientific Conference (2011, March 30)
The assessment of Collaborative Problem Solving in the context of MicroDYN and MicroFINScientific Conference (2011, January 30)
Geographische Kompetenzen fördern. Erfassung der Geographischen Systemkompetenz als Grundlage zur Bewertung der Kompetenzentwicklungin Meyer, C.; Henry, R.; Stöber, G. (Eds.) Geographische Bildung: Kompetenzen in didaktischer Forschung und Schulpraxis (2011)
Challenges of modern Computer Based Assessment: Usability, Scoring, and “Digital Natives”Scientific Conference (2011)
In recent years, computer based assessment has undergone substantive change. Test developers as well as test users have become aware of the fact that computers can do more than administrate traditional (paper-pencil) item formats like multiple-choice. More complex computer based item types allow tracking test takers’ mental representations of the problem or even their problem solving strategies by means of behavioral data. An example for such a modern item type are microworlds – dynamically changing problem solving scenarios with which the test taker has to interact. However, with the advent of complex item types, new challenges arise. First, usability is at stake - test takers do not intuitively know what to do or how to interact with complex tasks. Second, a massive load of data is produced and it gets difficult for the test developer to decide on relevant scores. Third, today’s students - so-called “digital natives”- grew up with computers and therefore set high quality standards for software applications. They may quickly loose trust and interest in tests with old fashioned design, cumbersome handling or even malfunctioning software. On basis of the Genetics Lab – a microworld developed to assess general mental ability – these challenges of modern computer based assessment are discussed. Three consecutive small scale studies were carried out to investigate usability issues, validate scoring algorithms and to ensure acceptance among students. The results demonstrate the importance of considering usability during the test development process, particularly with regard to scoring. The modification of conventional test development procedures for modern computer based assessment is suggested. Moreover, possibilities to satisfy even a critical target population are presented.
Dynamic systems and the theoretical concept of problem solving in PISA 2012Scientific Conference (2010, September 28)
Binnenstruktur und Validität: Minimal komplexe Probleme zur Diagnostik dynamischer ProblemlösefähigkeitScientific Conference (2010, September 26)
Dynamisches Problemlösen messbar gemacht: Der MicroDYN & MicroFIN-AnsatzPresentation (2010, June 08)
Measuring dynamic problem solving: Recent developments in MicroDYN and MicroFINScientific Conference (2010, May 22)
Dynamic Problem Solving: A New Perspective for Large-scale AssessmentsScientific Conference (2010, April 14)
Geographische Kompetenzen fördern. Erfassung der Geographischen Systemkompetenz als Grundlage zur Bewertung der KompetenzentwicklungScientific Conference (2010, March 18)
PISA switches to computer-based assessments: Chances that should not be missedPresentation (2010, February 10)
Systematische Erforschung komplexer Problemlösefähigkeit anhand minimal komplexer Systemein Zeitschrift für Pädagogik. Beiheft (2010), 56
Wandel im Bildungswesen, zunehmende Komplexität und Web 2.0: Sind traditionelle Intelligenztests noch "up to date"?Poster (2010)
Grundlegender Wandel im schulischen Bereich führt zur berechtigten Frage, inwiefern nicht Innovationen in der Intelligenzdiagnostik nötig wären, um die hohe prädiktive Validität und dadurch wesentliche Bedeutung bei Schulplatzierungsentscheidungen weiterhin zu sichern. Die stärkere Betonung fächerübergreifender Kompetenzen, eine vermehrte Einbindung des Computers in Unterricht und Alltag sowie zunehmende Komplexität in der Arbeitswelt führen zu neuen Herausforderungen und beleben alte Kritik wieder. In diesem Kontext wurden immer wieder computerbasierte Problemlöseszenarien als vielversprechender Ansatz genannt, Schwächen traditioneller Intelligenztests zu überwinden. Vorteile werden, neben dem dynamischen Testformat und einer vollständigeren Abdeckung des Intelligenzbegriffes, unter anderem auch in erhöhter face-validity gesehen. Während bisherige Studien den Schwerpunkt auf das empirische Verhältnis zwischen Leistungsmaßen der Szenarien und traditionellen Intelligenzmaßen setzten, versucht dieser Beitrag, bewusst die Diskussion um theoretische Aspekte zu ergänzen. Einer Analyse der Anforderungen die an eine aktuelle Intelligenzdiagnostik zu stellen sind, folgt ein systematischer Vergleich traditioneller Intelligenztestformate mit computerbasierten Problemlöseszenarien. Konkrete Beispiele sowie aktuelle Daten zur Pilotierung eines derartigen Szenarios zur Intelligenzdiagnostik ergänzen die Diskussion.
Three facets of complex problem solving: Information retrieval, model building & forecastingPresentation (2009, November 04)
COGSIM. Measuring general cognitive ability using computer-based complex problem solving scenariosScientific Conference (2009, September 06)
From process models to diagnostic practice: a unified approach to complex problem solvingScientific Conference (2009, August 25)
Difficult or not? Valid or not? A new way to measure complex problem solving: The MicroDYN approachScientific Conference (2009, August 24)
On the way to competence levels in dynamic microsystems: The MicroDYN ApproachScientific Conference (2009, August)
Trying to square a circle: A theoretically embedded test for measuring complex problem solvingScientific Conference (2009, August)
Assessing complex problem solving: German and international effortsScientific Conference (2009, April 26)
Problemlösen in der Experimentalpsychologie. Was wird eigentlich gemessen?Scientific Conference (2009, April 01)
Ein empirisch abgeleitetes Kompetenzmodell zur komplexen ProblemlösefähigkeitScientific Conference (2009, March 24)
Measuring Complex Problem Solving - The MicroDYN approachin Scheuermann, F.; Björnsson, J. (Eds.) The Transition to Computer-Based Assessment. Lessons learned from large-scale surveys and implications for testing (2009)
Complex problem as competence: Item difficulty and competence levels in dynamic situationsPoster (2008, July 22)
What makes a problem complex? Factors determining difficulty in dynamic situations and implications for diagnosing complex problem solving competencein Zumbach, Joerg; Schwartz, Neil; Seufert, Tina; Kester, Liesbeth (Eds.) Beyond knowledge: the legacy of competence (2008)
Indikatoren der Problemlöseleistung: Sinn und Unsinn verschiedener Berechnungsvorschriftenin Bericht aus dem MicroDYN Projekt (2008)
Collaborative Problem Solving in PISA 2015: Can ComputerAgents Replace Humans?in Computers in Human Behavior (n.d.)
Computer-Based Collaborative Problem Solving in PISA 2015 and the Role of the Big FiveDiverse speeches and writings (n.d.)
The Assessment of Collaborative Problem Solving in PISA 2015: An Investigation of the Validity of the PISA 2015 CPS Tasks.Diverse speeches and writings (n.d.)
