References of "Weinerth, Katja 40020922"
     in
Bookmark and Share    
See detailHow does usability improve computer-based knowledge assessment?
Weinerth, Katja UL

Doctoral thesis (2015)

There has been a major shift from paper-and-pencil towards computer-based assessments (CBAs). CBA has the potential to overcome various limitations imposed by traditional assessment approaches, mostly ... [more ▼]

There has been a major shift from paper-and-pencil towards computer-based assessments (CBAs). CBA has the potential to overcome various limitations imposed by traditional assessment approaches, mostly because CBA allows for the easier and more effective measurement of complex knowledge concepts via the use of dynamic items. Innovative item formats in CBA (dynamic or interactive multimedia items) allow the assessment of complex skills (e.g., complex problem solving), but they also tend to increase the complexity of CBA instruments because of their augmented interactivity. If the CBA is not user-friendly, the test-taker might spend more time and effort trying to understand how to interact with the system instead of focusing on the assessment task itself. The research field of human-computer interaction (HCI) shows that the usability or user-friendliness of the system affects the interaction. Usability addresses how appropriate (for a particular use) or how user-friendly a CBA instrument is. Thus, to the extent that there is any detrimental technical bias in CBA affecting the user-friendliness of a CBA, it will affect the user’s interaction with the instrument and consequently the instrument’s psychometrics. There are certain guidelines (e.g., International Test Commission, 2005) that require usability testing to safeguard against the CBA instrument measuring skills or competencies other than those that are supposed to be measured by the CBA instrument. However, surprisingly little research has been conducted to investigate whether usability has been acknowledged in CBAs and whether researchers are aware of the impact usability might have on the assessment results. To answer these and further research questions, three studies were conducted within this present Ph.D. project. These studies focused on a specific CBA instrument: concept maps. A concept map is a graphical illustration of a knowledge concept and can be described as a “knowledge net.” Research has shown that the concept map is a valid and reliable instrument for measuring conceptual knowledge. Concept map instruments are a good example of the increasing popularity of CBA, as technology allows test-takers to correct and improve their concept maps with ease. Test-takers and their examiners receive immediate feedback on the correctness of the concept map. Furthermore, it is a good example of new approaches to knowledge assessment, which is currently a major trend. All of this taken together makes concept maps a good proxy for new ways of assessing knowledge. Our research builds on three studies (Study I – literature-based study, Study II – laboratory study, Study III – school study) that investigated the impact of usability on computer-based knowledge assessment using concept maps. Study I, a literature-based study, scrutinized the existing literature and answered the fundamental research question of whether and how usability has been acknowledged in CBA concept map studies. As no such literature review existed prior to the current project, a systematic literature review was conducted to shed light on the representation and relevance of usability displayed in CBA concept map studies. The literature review indicated that only 24 of 119 journal articles that assessed computer-based concept maps actually discussed the usability of the applied instrument in some way and only three of those 24 journal articles explicitly mentioned that they evaluated the usability of the applied instruments. The literature review illustrated that usability is rarely acknowledged and reported in CBA concept map studies. Our review brings to light the idea that the impact of usability, although well established in the field of HCI, has received insufficient attention in the field of educational assessment. Study II, a laboratory study, addressed the main research question of how HCI methods can be introduced to the field of educational assessment to improve the usability of a CBA concept map instrument. With a user-centered design and development approach, which makes the user the key reference, usability testing and heuristic analyses were conducted in the usability laboratory. The approved HCI methods of usability testing and heuristic analyses were combined to evaluate and further improve a CBA concept map instrument. We applied three iterative design and re-engineering cycles that were based on the results we received from the usability testing and heuristic analyses. To verify the improvements in the usability of the CBA concept map instrument, three independent and randomly assigned groups of 30 students underwent concept map assessments using the three iteratively developed instruments (baseline (V1), further developed: V2 and V3). The results of this independent design sample study showed that the HCI methods allowed us to design and develop demonstrably usability-improved concept map instruments; they furthermore revealed that the usability improvements significantly improved the assessment outcomes. Study III, a school study, empirically verified the impact that usability has on the psychometrics of the applied CBA instruments and on the satisfaction and performance of the test-takers. The experimental study was conducted at school, and 542 students were randomly assigned to one of the three CBA concept map instruments that showed consecutively improved usability. The performance of the test-takers who worked with the usability-improved instruments significantly increased in comparison with the baseline version. The test-takers indicated that they were more satisfied with the usability-improved instruments. Moreover, the internal consistency of the items from the baseline instrument in comparison with usability-improved instruments increased from a Cronbach’s alpha of .62 to .84. To summarize, Study I showed that the impact of usability is rarely evaluated and discussed in CBA concept map studies. Studies II and III clearly showed that usability has a positive impact on the test-takers’ interaction with the CBA concept map instrument. Specifically, Study II illustrated and demonstrated how HCI methods could be used to achieve usability-improved CBA instruments, which in turn allow for better assessment outcomes. In addition, Study III illustrated that the psychometrics were also affected by the usability of the instrument; specifically, the reliability (measured as the internal consistency of the applied items) increased when the usability-improved instruments were used. Thus, the continuing trend towards CBAs calls for more systematic usability research to help ensure satisfactory psychometric test properties and test-takers’ satisfaction with the instrument. The studies confirmed the hypothesis that if usability is not taken into account, the assessment results may severely compromise the quality of individual diagnostics as well as educational policies and educational decisions. [less ▲]

Detailed reference viewed: 155 (34 UL)
Full Text
Peer Reviewed
See detailConcept maps: A useful and usable tool for computer-based knowledge assessment? A literature review with a focus on usability
Weinerth, Katja UL; Koenig, Vincent UL; Brunner, Martin UL et al

in Computers & Education (2014), 78

The concept map is now widely accepted as an instrument for the assessment of conceptual knowledge and is increasingly being embedded into technology-based environments. Usability addresses how ... [more ▼]

The concept map is now widely accepted as an instrument for the assessment of conceptual knowledge and is increasingly being embedded into technology-based environments. Usability addresses how appropriate (for a particular use) or how user-friendly a computer-based assessment instrument is. As we know from human-computer interaction research, if the interface is not user-friendly, a computer-based assessment can result in decreased test performance and reduced validity. This means that the usability of the interface affects the assessment in such a way that if the test is not user-friendly, then the test taker will not be able to fully demonstrate his/her level of proficiency and will instead be scored according to his/her information and communication technology (ICT) literacy skills. The guidelines of the International Test Commission (2006) require usability testing for such instruments and suggest that design standards be implemented. However, we do not know whether computer-conducted concept map assessments fulfill these standards. The present paper addresses this aspect. We conducted a systematic research review to examine whether and how researchers have studied and considered usability when conducting computer-based concept map assessments. Only 24 out of 119 journal articles that assessed computer-based concept maps discussed the usability issue in some way. Nevertheless, our review brings to light the idea that the impact of usability on computer-based concept map assessments is an issue that has received insufficient attention. In addition, usability ensures a suitable interaction between test taker and test device; thus, the training effort required for test use can be reduced if a test's usability is straight forward. Our literature review, however, illustrates that the interplay between usability and test use training has mostly been neglected in current studies. [less ▲]

Detailed reference viewed: 225 (25 UL)
Full Text
Peer Reviewed
See detailHCI Design Methods in CBA: Lessons Learned and Recommendations Based on a Concept Map Use Case
Weinerth, Katja UL; Koenig, Vincent UL; Rivas, Salvador UL et al

in Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2014 (2014)

This paper focuses on the user-centered design of a computer-based assessment (CBA) instrument for knowledge measurement. We describe relevant design steps in general and illustrate them via the design of ... [more ▼]

This paper focuses on the user-centered design of a computer-based assessment (CBA) instrument for knowledge measurement. We describe relevant design steps in general and illustrate them via the design of our usability improved CBA concept map instrument (ISO 9241-11, 1998; ISO 9241-210, 2010). We discuss design versus evaluation methods from a human-computer interaction (HCI) perspective and emphasize important design steps relevant for constructing usability improved assessment instruments in educational assessment. As these HCI methods are still seldom applied to CBA, this paper helps alleviate the dearth of literature and guidelines in this domain. Our objective is thus to provide the reader with a series of design recommendations for CBA, as derived both from HCI research; and to summarize lessons learned from our specific CBA design use case. [less ▲]

Detailed reference viewed: 124 (15 UL)
See detailThe importance of Human-Computer Interaction in Computer-Based Assessment
Martin, Romain UL; Koenig, Vincent UL; Weinerth, Katja UL

Scientific Conference (2013, October 01)

Detailed reference viewed: 78 (15 UL)
See detailImpact of Usability on Computer-Based Concept Maps in Educational Assessment
Weinerth, Katja UL; Koenig, Vincent UL; Brunner, Martin et al

Presentation (2013, July 03)

Detailed reference viewed: 56 (7 UL)
Full Text
Peer Reviewed
See detailEducational Assessment Meets Usability: Equal Chances for Conducting Computer-Based Concept-Maps
Weinerth, Katja UL; Koenig, Vincent UL; Brunner, Martin et al

in Herrington, Jan; Couros, Alec; Irvine, Valerie (Eds.) EdMedia World Conference on Educational Media & Technology 2013 (2013)

This paper describes how usability affects the usage of a computer-based assessment (CBA) instrument. The aim of our current study is twofold: a) the development of a usability improved CBA instrument ... [more ▼]

This paper describes how usability affects the usage of a computer-based assessment (CBA) instrument. The aim of our current study is twofold: a) the development of a usability improved CBA instrument (ISO 9241-11, 1998; ISO 9241-210, 2010) within the framework of human-computer interaction (HCI) and b) the comparison and evaluation of two concept map applications in regards to the impact that usability has on the assessment interaction. In this study we developed and evaluated two CBA concept map applications. We assessed 60 students aged between 16 and 18 who were asked to conduct concept maps on two different versions of the same knowledge assessment instrument. We observed significant differences regarding two out of three usability criteria of the ISO standard definition: efficiency and satisfaction of the students. Our findings highlight that usability influences the assessment interaction and therefore the chances of the assessment taker to work (successfully) through the assessment. [less ▲]

Detailed reference viewed: 94 (10 UL)
Full Text
Peer Reviewed
See detailL'évaluation des compétences scientifiques au 21eme siècle : cadre conceptuel et prototypes pour une évaluation transformée-par-ordinateur
Weinerth, Katja UL; Reuter, Robert UL; Porro, Vincent et al

in Actes du 24e colloque international de l'ADMEE-Europe (2012)

En parallèle à l’évolution récente de l’enseignement des sciences (vers des modes d’enseignement basée sur la découverte guidée ou autonome), on observe une course aux nouvelles méthodes d’évaluation des ... [more ▼]

En parallèle à l’évolution récente de l’enseignement des sciences (vers des modes d’enseignement basée sur la découverte guidée ou autonome), on observe une course aux nouvelles méthodes d’évaluation des compétences scientifiques. Nous proposons ici un cadre conceptuel que nous avons développé et que nous utilisons actuellement pour guider le développement d’un environnement informatique d’apprentissage et d’évaluation intégré, qui pourra être utilisé (1) pour offrir aux apprenants une large panoplie de situations d’apprentissage relativement authentiques, implémentant l’ensemble des activités de l’investigation scientifique et (2) pour évaluer leurs connaissances du processus d’investigation scientifique, pour évaluer leurs compétences « en action » et pour évaluer comment ils appliquent leurs connaissances d’un domaine scientifique donné. Ce cadre conceptuel sera illustré à l’aide de prototypes d’items d’évaluation développés pour des ordinateurs mobiles à écran tactile. [less ▲]

Detailed reference viewed: 78 (12 UL)
Full Text
See detailUsability meets assessment: Equal chances in education
Weinerth, Katja UL; Koenig, Vincent UL

in World Usability Day 2011 (2011)

Detailed reference viewed: 51 (6 UL)