Experimenter Effects in Children Using the Smileyometer Scale

Researchers in the social sciences like human-computer interaction face novel challenges concerning the development of methods and tools for evaluating interactive technology with children. One of these challenges is related to the validity and reliability of user experience measurement tools. Scale designs, like the Smileyometer, have been proven to contain biases such as the tendency for children to rate almost every technology as great. This explorative paper discusses a possible effect of two experimenter styles on the distribution of 6-8 years old pupils' ratings (N= 73) to the Smileyometer. We administered the scale before and after a tablet-based assessment in two schools. Experimenter 1 employed a child-directed speech compared to a monotone speech of Experimenter 2. While brilliant (5 out of 5) was the most frequent answer option in all conditions, the mean scores were higher and associated with a lower variability across both conditions for Experimenter 2. We discuss a possible experimenter effect in the Smileyometer and implications for evaluating children’s user experiences.