Construct validity of complex problem solving: A comprehensive view on different facets of intelligence and school grades

in Intelligence (2016), 54

Although Complex Problem Solving (CPS) has attracted increasing amounts of attention in recent years (e.g., PISA study), the role of CPS in the nomological network of intelligence is controversial. The ... [more ▼]

Although Complex Problem Solving (CPS) has attracted increasing amounts of attention in recent years (e.g., PISA study), the role of CPS in the nomological network of intelligence is controversial. The question of whether CPS is a distinct construct is as old as CPS research itself, but previous studies have had specific shortcomings when addressing the question of whether CPS is a separable or independent construct. The aim of the present study was, therefore, to combine the advantages of previous studies to facilitate a less biased view of the relation between CPS and established intelligence constructs. A sample of 227 German university students worked on a comprehensive measure of intelligence (Berlin Intelligence Structure test) and two CPS assessment tools (MicroDYN and MicroFIN). Furthermore, final school grades (GPA) served as an external criterion. We applied confirmatory factor analyses and structural equation modeling to investigate the relation between CPS and established intelligence constructs on the basis of different psychometric approaches (i.e., first-order model, nested factor model). Moreover, we examined the incremental validity of CPS in explaining GPA beyond established intelligence constructs. Results indicate that CPS represents unique variance that is not accounted for by established intelligence constructs. The incremental validity of CPS was found only when a commonly used narrow operationalization of intelligence was applied (i.e., figural reasoning) but not when a broad operationalization was applied. [less ▲]

Determinants of individual occupational careers in the 21st century. Does complex problem solving matter beyond general mental ability?

Scientific Conference (2015, May 21)

Purpose Complex problem solving (CPS) describes the interaction with dynamic and nonroutine tasks, and has been included in PISA 2012 as a factor for employability. This study examines whether CPS can ... [more ▼]

Purpose Complex problem solving (CPS) describes the interaction with dynamic and nonroutine tasks, and has been included in PISA 2012 as a factor for employability. This study examines whether CPS can also contribute to the prediction of career advancement in jobs beyond general mental ability (GMA) as one of the best predictors. Design/Methodology Using latent structural equation modeling (SEM), we analyzed a sample of technicians, service/trade workers, and assemblers (n=245) at a German automotive company. A computer-based assessment measured participants' CPS and GMA levels. The dependent variables were the participants’ job level (ISCO-08) and professional training days. Results CPS and GMA both correlated significantly with career advancement (from .18 to .26, all p < .01). The models showed good fit and indicated that CPS explained incremental variance in one of two indicators (β = .14 for trainings, p < .05; ΔR2 = .02) in comparison with GMA alone (β = .24, p < 0.01; R2= .06). Limitations Analyses did not include processes information from CPS assessment as potential advantage. The company-based sample and cross-sectional data restrict inferences. Research Implications Our findings suggest positive relations between CPS and career advancement even when controlling for GMA. Hence, CPS could be a valuable addition for the study of careers and personnel selection test batteries. Originality/Value The first evaluation of CPS in career research gave a general indication of an as-yet-to-be-defined role of CPS, especially when considering the task characteristics compared to complex and demanding jobs, and process data available through CPS. [less ▲]

Thematic workshop on complex problem solving.

Scientific Conference (2015)

Extending the assessment of Complex Problem Solving to finite state automata. Embracing heterogeneity

in European Journal of Psychological Assessment (2015), 31

The Assessment of 21st Century Skills in Industrial and Organizational Psychology: Complex and Collaborative Problem Solving

in Industrial and Organizational Psychology (2015), 8(2), 1-31

In the current paper, we highlight why and how industrial and organizational psychology can take advantage of research on 21st century skills and their assessment. We present vital theoretical ... [more ▼]

In the current paper, we highlight why and how industrial and organizational psychology can take advantage of research on 21st century skills and their assessment. We present vital theoretical perspectives, a suitable framework for assessment, and exemplary instruments with a focus on advances in the assessment of Human Capital. Specifically, Complex Problem Solving (CPS) and Collaborative Problem Solving (ColPS) are two transversal skills (i.e., skills that span multiple domains) that are generally considered critical in the 21st century workplace. The assessment of these skills in education has linked fundamental research with practical applicability and has provided a useful template for workplace assessment. Both CPS and ColPS capture the interaction of individuals with problems that require the active acquisition and application of knowledge in individual or group settings. To ignite a discussion in industrial and organizational psychology, we discuss advances in the assessment of CPS and ColPS and propose ways to move beyond the current state of the art in assessing job-related skills. [less ▲]

Progress in the LLLight project. Report from the assessment team from Luxembourg.

Presentation (2014, October)

Determinants of Individual Occupational Careers in the 21st Century – Why Complex Problem Solving Matters Beyond General Mental Ability

Scientific Conference (2014, August 27)

The OECD (2010) views problem solving as essential for lifelong learning. Recently, the 2012 PISA cycle administered domain-general problems, which featured the complexity of the structure, dynamics and ... [more ▼]

The OECD (2010) views problem solving as essential for lifelong learning. Recently, the 2012 PISA cycle administered domain-general problems, which featured the complexity of the structure, dynamics and interconnections of variables, and intransparency of the situation. This emphasis on domain-general problems expresses a paradigmatic shift to lifelong learning. In an attempt towards an integrative process theory of CPS, Fischer, Greiff and Funke (2012) portray knowledge acquisition and knowledge application as the core processes of CPS, which are distinguishable from classical cognitive ability (Dörner, 1986). These core processes are assessed in research via innovative computer-based microworlds. These microworlds simulate complex problems, which share features with real world problems, require knowledge acquisition and knowledge application for a successful task completion and offer a wide range of tasks and approaches (Greiff, Holt, et al., 2013; Greiff, Fischer, et al., 2013). Initiatives looking for the facilitation of so-called 21st century skills (e.g., Griffin, McGaw, & Care, 2012; National Research Council, 2012a; OECD, 2013a, 2013b) strongly relate CPS to demands for professional development as resulting from shifts towards non-routine problem solving and communication tasks in the working lives of adults (e.g., successfully adapting towards IT-depending occupations; Autor, Levy, & Murnane, 2003). So far, the application of CPS in research and practice has mainly been restricted to primary and secondary education. Lifespan or educational research on the role of transversal cognitive skills such as CPS for learning later in life is scarce at best and adult education practice fails to fully tap into the requirements of a working reality (Smith & Reio, 2006), which is defined by uncertainties, contradictions, and dynamic changes. CPS is considered to be important for lifelong learning and occupational success, but underrepresented in research and practice. As a consequence, adult education’s mission to provide opportunities for career preparation (e.g., Gianakos, 1996) and intellectual growth (Perry, 1999) seem to be at risk. Hence, the research goals in this study are to (1) enlarge the understanding for determinants of occupational career indicators (e.g. lifelong learning efforts) and eventually (2) encourage the inclusion of CPS on the adult educational research and practice agenda. Our research goals are built on a rich research fundus of the impact of GMA on professional development (e.g., Schmidt & Hunter, 1998) and promising results on the predictive validity of CPS for academic achievement (e.g., Schweizer, Wüstenberg, & Greiff, 2013; Sonnleitner, Keller, Martin, & Brunner, 2013; Wüstenberg, Greiff, & Funke, 2012) and occupational performance (Abele et al., 2012; Danner et al., 2011). Hence, the targeted constructs are GMA, and CPS as a transversal cognitive skill in occupational careers and professional development. [less ▲]

Political implications of educational large scale assessments on the example of complex problem solving.

Scientific Conference (2014, April 30)

Computer-based assessment of Complex Problem Solving in educational contexts and how it is influenced by students’ level of Information and Communication Technology literacy

in Journal of Educational Psychology (2014)

Extending the Assessment of Complex Problem Solving to Finite State Automata: Embracing Heterogeneity

in European Journal of Psychological Assessment (2014), Advance Online Publication

Recent advancements in the assessment of Complex Problem Solving (CPS) build on the use of homogenous tasks that enable the reliable estimation of CPS skills. The range of problems featured in established ... [more ▼]

Recent advancements in the assessment of Complex Problem Solving (CPS) build on the use of homogenous tasks that enable the reliable estimation of CPS skills. The range of problems featured in established instruments such as MicroDYN is consequently limited to a specific subset of homogeneous complex problems. This restriction is problematic when looking at domain-specific examples of complex problems, which feature characteristics absent from current assessment instruments (e.g., threshold states). We propose to utilize the formal framework of Finite State Automata (FSA) to extend the range of problems included in CPS assessment. An approach based on FSA, called MicroFIN, is presented, translated into specific tasks, and empirically investigated. We conducted an empirical study (N = 576), (1) inspecting the psychometric features of MicroFIN, (2) relating it to MicroDYN, and (3) investigating the relations to a measure of reasoning (i.e., CogAT). MicroFIN (1) exhibited adequate measurement characteristics and multitrait- multimethod models indicated (2) the convergence of latent dimensions measured with MicroDYN. Relations to reasoning (3) were moderate and comparable to the ones previously found for MicroDYN. Empirical results and corresponding explanations are discussed. More importantly, MicroFIN highlights the feasibility of expanding CPS assessment to a larger spectrum of complex problems. [less ▲]