Challenges of modern Computer Based Assessment: Usability, Scoring, and “Digital Natives”

In recent years, computer based assessment has undergone substantive change. Test developers as well as test users have become aware of the fact that computers can do more than administrate traditional (paper-pencil) item formats like multiple-choice. More complex computer based item types allow tracking test takers’ mental representations of the problem or even their problem solving strategies by means of behavioral data. An example for such a modern item type are microworlds – dynamically changing problem solving scenarios with which the test taker has to interact.
However, with the advent of complex item types, new challenges arise. First, usability is at stake - test takers do not intuitively know what to do or how to interact with complex tasks. Second, a massive load of data is produced and it gets difficult for the test developer to decide on relevant scores. Third, today’s students - so-called “digital natives”- grew up with computers and therefore set high quality standards for software applications. They may quickly loose trust and interest in tests with old fashioned design, cumbersome handling or even malfunctioning software.
On basis of the Genetics Lab – a microworld developed to assess general mental ability – these challenges of modern computer based assessment are discussed. Three consecutive small scale studies were carried out to investigate usability issues, validate scoring algorithms and to ensure acceptance among students. The results demonstrate the importance of considering usability during the test development process, particularly with regard to scoring. The modification of conventional test development procedures for modern computer based assessment is suggested. Moreover, possibilities to satisfy even a critical target population are presented.