The assessment of collaborative problem solving in PISA 2015: An investigation of the validity of the PISA 2015 CPS tasks

in Computers and Education (2020)

Students' exploration strategies in computer-simulated complex problem environments: A latent class approach

in Computers and Education (2018), 126

Complex problem solving (CPS) is considered an important educational achievement indicator. Previous research has indicated that CPS performance depends to a substantial extent on the way students explore ... [more ▼]

Complex problem solving (CPS) is considered an important educational achievement indicator. Previous research has indicated that CPS performance depends to a substantial extent on the way students explore problem environments. In this study, we investigated qualitative differences in the way students interact with such environments. In a sample of N = 2226 Hungarian students in Grades 6 to 8, we applied a latent class approach to investigate the use of the principle of isolated variation as an exploration strategy across six CPS tasks that were developed within the MicroDYN approach. Six qualitatively different class profiles emerged: proficient explorers, intermediate explorers, low-performing explorers, rapid learners, emerging explorers, and nonpersisting explorers. We further validated the profiles by comparing the latent classes with regard to students' overall CPS performance and additional indicators of task exploration. In analyzing age-related and gender differences on a cross-sectional level, we found only a small progression toward better performing class profiles from Grade 6 to Grade 8 (e.g., 14.6% of students in Grade 6 were proficient explorers vs. 24.6% in Grade 8; 27.1% of students in Grade 6 were low-performing explorers vs. 25.8% in Grade 8), and there were no substantial gender differences. This study contributes to the understanding of how students interact with complex problems and is the first to address whether variations in these behaviors indicate qualitatively different levels of strategic behavior. We discuss the theoretical underpinnings and potential of identifying class profiles of students' exploration behavior in the field of educational psychology. [less ▲]

Concept maps: A useful and usable tool for computer-based knowledge assessment? A literature review with a focus on usability

in Computers and Education (2014), 78

The concept map is now widely accepted as an instrument for the assessment of conceptual knowledge and is increasingly being embedded into technology-based environments. Usability addresses how ... [more ▼]

The concept map is now widely accepted as an instrument for the assessment of conceptual knowledge and is increasingly being embedded into technology-based environments. Usability addresses how appropriate (for a particular use) or how user-friendly a computer-based assessment instrument is. As we know from human-computer interaction research, if the interface is not user-friendly, a computer-based assessment can result in decreased test performance and reduced validity. This means that the usability of the interface affects the assessment in such a way that if the test is not user-friendly, then the test taker will not be able to fully demonstrate his/her level of proficiency and will instead be scored according to his/her information and communication technology (ICT) literacy skills. The guidelines of the International Test Commission (2006) require usability testing for such instruments and suggest that design standards be implemented. However, we do not know whether computer-conducted concept map assessments fulfill these standards. The present paper addresses this aspect. We conducted a systematic research review to examine whether and how researchers have studied and considered usability when conducting computer-based concept map assessments. Only 24 out of 119 journal articles that assessed computer-based concept maps discussed the usability issue in some way. Nevertheless, our review brings to light the idea that the impact of usability on computer-based concept map assessments is an issue that has received insufficient attention. In addition, usability ensures a suitable interaction between test taker and test device; thus, the training effort required for test use can be reduced if a test's usability is straight forward. Our literature review, however, illustrates that the interplay between usability and test use training has mostly been neglected in current studies. [less ▲]