Sentence Completion as a User Experience Research Method: Recommendations from an Experimental Study

Schmidt, Denise; Nebe Karsten; LALLEMAND, Carine

doi:10.1093/iwc/iwae002

Request a copy

Article (Scientific journals)

Sentence Completion as a User Experience Research Method: Recommendations from an Experimental Study

Schmidt, Denise; Nebe Karsten; LALLEMAND, Carine

2024 • In Interacting with Computers

Peer Reviewed verified by ORBi Dataset

Permalink
https://hdl.handle.net/10993/60536

DOI
10.1093/iwc/iwae002

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

iwae002.pdf

Publisher postprint (986.57 kB)

Request a copy

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

sentence completion; method; user research; design; online survey; survey technique; User experience; UX; tool

Abstract :

[en] The aim of this study is to investigate the use of the sentence completion technique (SCT) as a user experience (UX) research method. We conducted an online experiment (N = 400) to test the effect of sentence stem variations on sentence completion outcomes. Using a between-subjects design, half of the participants were exposed to impersonal sentence stems that did not include pronouns while the other half were exposed to stems formulated using first-person pronouns (PR). Additional hypotheses around stem formats (use of redundant stems, imaginative stems, two blanks stems, generic stems, stems prompting others’ perception) were tested using a within-subject design. The results do not support hypothesized differences between the pronoun and no pronoun condition. Findings however show that varying the format of the stem influences response behavior, as measured by variety, quantity and novelty of ideas, as well as the length of response. This study contributes to consolidating the use of SCT as a user research method and proposes actionable recommendations on how to create optimal sentence completion surveys in Human-Computer Interaction (HCI).

Disciplines :

Social & behavioral sciences, psychology: Multidisciplinary, general & others

Author, co-author :

Schmidt, Denise ^✱

Nebe Karsten

LALLEMAND, Carine ^✱; University of Luxembourg > Faculty of Humanities, Education and Social Sciences (FHSE) > Department of Behavioural and Cognitive Sciences (DBCS) > Cognitive Science and Assessment ; Eindhoven University of Technology > Industrial Design

^✱ These authors have contributed equally to this work.

External co-authors :

yes

Language :

English

Title :

Sentence Completion as a User Experience Research Method: Recommendations from an Experimental Study

Publication date :

06 March 2024

Journal title :

Interacting with Computers

ISSN :

0953-5438

eISSN :

1873-7951

Publisher :

Oxford University Press, Oxford, United Kingdom

Peer reviewed :

Peer Reviewed verified by ORBi

Data Set :

10.1093/iwc/iwae002

Available on ORBilu :

since 09 March 2024

Statistics

Number of views

88 (0 by Unilu)

Number of downloads

0 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

Bargas-Avila, J. A. and Hornbæk, K. (2011). Old Wine in New Bottles or Novel Challenges? A Critical Analysis of Empirical Studies of User Experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). Association for Computing Machinery, New York, NY, USA, pp. 2689–2698. https://doi.org/10.1145/1978942.1979336
Behrend, T. S., Sharek, D. J., Meade, A. W. and Wiebe, E. N. (2011) The viability of crowdsourcing for survey research. Behav. Res. Methods, 43, 800–813. https://doi.org/10.3758/s13428-011-0081-0.
Berinsky, A. J., Huber, G. A. and Lenz, G. S. (2012) Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Polit. Anal., 20, 351–368. https://doi.org/10.1093/pan/mpr057.
Cockton, G., Kujala, S., Nurkka, P. and Hölttä, T. (2009). Supporting Worth Mapping with Sentence Completion. In Gross, T., et al. (eds.), Human-Computer Interaction – INTERACT 2009. INTERACT 2009. Lecture Notes in Computer Science, vol 5727. Springer, Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-03658-3_61
Cromwell, R. L. and Lundy, R. M. (1954) Productivity of clinical hypotheses on a sentence completion test. J. Consult. Psychol., 18, 421–424. https://doi.org/10.1037/h0059972.
Distler, V., Fassl, M., Habib, H., Krombholz, K., Lenzini, G., Lallemand, C., Cranor, L. F. and Koenig, V. (2021) A systematic literature review of empirical methods and risk representation in usable privacy and security research. ACM Trans. Comput.-Hum. Interact., 28, 50. https://doi.org/10.1145/3469845.
Donoghue, S. (2010) Projective techniques in consumer research. J. Fam. Ecol. Consum. Sci., 28, 47–53. https://doi.org/10.4314/jfecs.v28i1.52784.
Ebbinghaus, H. Z. (1897) Uber eine neue methode zur prüfung geistiger fähigkeiten and ihre anwendung bei schulkindern [About a new method for testing intellectual abilities]. Psychologie Und Physiologie Der Sinnesorgane, 13, 401–457.
Fitt, H. (2018) Researching mobile practices: participant reflection and audio-recording in repeat question diaries. Qual. Res., 18, 654–670. https://doi.org/10.1177/1468794117743462.
Goelema, M. S., de Bruijn, R., Overeem, S., Møst, E., Haakma, R. and Markopoulos, P. (2018) Conceptions of sleep experience: a layman perspective. BMC Res. Notes, 11, 1–5. https://doi.org/10.1186/ s13104-018-3584-2.
Goldberg, P. A. (1965) A review of sentence completion methods in personality assessment. J. Proj. Tech. Pers. Assess., 29, 12–45. https://doi.org/10.1080/0091651X.1965.10120175.
Helkiö, L., Hölttä, T., Kujala, S. and Nurkka, P. (2010) Brand as a Basis for Cross-Cultural User Experience Design. In Proceedings of the Ninth International Workshop on Internationalisation of Products and Systems IWIPS 2010, 7–10 July 2010. Designing for Global Markets, London, UK, pp. 11–22.
Holaday, M., Smith, D. A. and Sherry, A. (2000) Sentence completion tests: a review of the literature and results of a survey of members of the Society for Personality Assessment. J. Pers. Assess., 74, 371–383. https://doi.org/10.1207/S15327752JPA7403_3.
Hoyer, W. D. and MacInnis, D. J. (2006) Consumer behavior (4th). Houghton Mifflin College Div, Boston.
Hsieh, H.-F. and Shannon, S. E. (2005) Three approaches to qualitative content analysis. Qual. Health Res., 15, 1277–1288. https://doi.org/10.1177/1049732305276687.
Kao, C.-Y. (2016) The effects of stimulus words’ positions and properties on response words and creativity performance in the tasks of analogical sentence completion. Learn. Individ. Differ., 50, 114–121. https://doi.org/10.1016/j.lindif.2016.07.015.
Kohli, A., Jhanda, S. and Padhy, S. (2018) The utility of modified version of sentence completion test for children and adolescents. Ind. Psychiatry J., 27, 231–234. https://doi.org/10.4103/ipj.ipj_63_18.
Kujala, S. and Nurkka, P. (2009). Identifying user values for an activating game for children. In Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era (MindTrek’09). Association for Computing Machinery, New York, NY, USA, pp. 98–105 . https://doi.org/10.1145/1621841.1621860
Kujala, S. and Nurkka, P. (2012) Sentence completion for evaluating symbolic meaning. Int. J. Des., 6, 15–25.
Kujala, S., Walsh, T., Nurkka, P. and Crisan, M. (2014) Sentence completion for understanding users and evaluating user experience. Interact. Comput., 26, 238–255. https://doi.org/10.1093/iwc/ iwt036.
Lallemand, C. (2015) Towards Consolidated Methods for the Design and Evaluation of User Experience [Doctoral dissertation]. University of Luxembourg, Luxembourg.
Lallemand, C. and Gronier, A. G. (2018) Méthodes de design UX: 30 méthodes fondamentales pour concevoir des expériences optimales. Eyrolles, Paris.
Lallemand, C. and Koenig, V. (2017). “How could an intranet be like a friend to me?” - Why standardized UX scales don’t always fit. In Proceedings of the European Conference on Cognitive Ergonomics (ECCE ’17). Association for Computing Machinery, New York, NY, USA, pp. 9–16. https://doi.org/10.1145/3121283.3121288
Lallemand, C. and Mercier, E. (2022). Optimizing the Use of the Sentence Completion Survey Technique in User Research: A Case Study on the Experience of E-Reading. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI’22). Association for Computing Machinery, New York, NY, USA, Article 319, pp. 1–18. https://doi.org/10.1145/3491102.3517718
Meltzoff, J. (1951) The effect of mental set and item structure upon response to a projective test. J. Abnorm. Soc. Psychol., 46, 177–189. https://doi.org/10.1037/h0058624.
Menheere, D., Funk, M., van der Spek, E., Lallemand, C. and Vos, S. (2020). A Diary Study on the Exercise Intention-Behaviour Gap: Implications for the Design of Interactive Products. In Boess, S., Cheung, M. and Cain, R. (eds.), Synergy - DRS International Conference 2020, 11-14 August. Held online. https://doi.org/10.21606/drs.2020.329
Murstein, B. I., Colon, R. M., Leval, N. D. and van Parys, M. H.-V. (1972) Inf luence of stimulus properties of the sentence-completion-method on projection and adjustment. J. Pers. Assess., 36, 241–247. https://doi.org/10.1080/00223891.1972.10119751.
Nakamura, W. T., Ahmed, I., Redmiles, D., Oliveira, E., Fernandes, D., de Oliveira, E. H. T. and Conte, T. (2021) Are UX evaluation methods providing the same big picture? Sensors, 21, 3480. https://doi.org/10.3390/s21103480.
Ninomiya, Y. et al. (2021) A cross-cultural study of happiness in Japanese, Finnish, and Mongolian children: analysis of the sentence completion test. Child Indic. Res., 14, 871–896. https://doi.org/10.1007/s12187-020-09776-y.
Nurkka, P., Kujala, S. and Kemppainen, K. (2009) Capturing users’ perceptions of valuable experience and meaning. J. Eng. Des., 20, 449–465. https://doi.org/10.1080/09544820903158835.
Peck, R. F. and Carson, M. G. (1959) Measuring changes in mental health with the sentence completion technique. Psychol. Rep., 5, 151–160. https://doi.org/10.2466/pr0.1959.5.g.151.
Peer, E., Rothschild, D., Gordon, A., Evernden, Z. and Damer, E. (2022) Data quality of platforms and panels for online behavioral research. Behav. Res. Methods, 54, 1643–1662. https://doi.org/10.3758/s13428-021-01694-3.
Petrie, H., Walsh, T., Odutola, O. and Ang, L. (2015) Cross-cultural issues in working with users in the design of interactive systems. ICoRD’15 Smart Innov. Syst. Technol., 34, 515–525. https://doi.org/10.1007/978-81-322-2232-3_45.
Piotrowski, C. (2018) Sentence Completion Methods: A Summary Review of 70 Survey-based Studies of Training and Professional Settings. ISIS J. Proj. Psy. & Ment., 25, 60–75.
Raffaelli, M., Koller, S. H., Reppold, C. T., Kuschick, M. B., Krum, F. and Bandeira, D. (2001) How do Brazilian Streeth youth experience the street? Analysis of a sentence completion task. Childhood, 8, 396–415. https://doi.org/10.1177/0907568201008003006.
Rees, A. and Nicholson, N. (2004). The Twenty Statements Test. In Cassell, C. and Symon, G. (eds.), Essential Guide to Qualitative Methods in Organizational Research. SAGE Publications Ltd., London, pp. 86–97. https://doi.org/10.4135/9781446280119.n8
Ridgeway, C. (2017). Projective Measures and Occupational Assessment. In B. Cripps (Ed.), Psychometric Testing: Critical Perspectives. John Wiley & Sons, New York. https://doi.org/10.1002/9781119183020.ch15.
Robinson, J., Lanius, C. and Weber, R. (2018) The past, present, and future of UX empirical research. Communic. Des. Quart., 5, 10–23. https://doi.org/10.1145/3188173.3188175.
Rogers, K. E., Bishop, J. and Lane, R. C. (2003) Considerations for the use of sentence completion tests. J. Contemp. Psychother., 33, 235–242. https://doi.org/10.1023/A:1023977704349.
Rohles, B., Koenig, V., Fischbach, A. and Amadieu, F. (2019) Experience matters: bridging the gap between experience- and functionality-driven design in technology-enhanced learning. Interact. Des. Architect., 42, 11–28. https://doi.org/10.55612/s-5002-042-001.
Roto, V., Law, E., Vermeeren, A. and Hoonhout, J. (eds.) (2011). User Experience White Paper: Bringing clarity to the concept of user experience. In Outcome of Dagstuhl Seminar 10373 on Demarcating User Experience. https://experienceresearchsociety.org/wpcontent/uploads/2023/01/UX-WhitePaper.pdf
Rotter, J. B. and Willerman, B. (1947) The incomplete sentences test as a method of studying personality. J. Consult. Psychol., 11, 43–48. https://doi.org/10.1037/h0059479.
Sacks, J. M. (1949) The relative effect upon projective responses of stimuli referring to the subject and of stimuli referring to other persons. J. Consult. Psychol., 13, 12–20. https://doi.org/10.1037/ h0055829.
Shah, J. J., Vargas-Hernandez, N. and Smith, S. M. (2003) Metrics for measuring ideation effectiveness. Des. Stud., 24, 111–134. https://doi.org/10.1016/S0142-694X(02)00034-0.
Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T., Hess, K. I., Martinez, J. L. and Richard, C. A. (2008) Assessing creativity with divergent thinking tasks: exploring the reliability and validity of new subjective scoring methods. Psychol. Aesthet. Creat. Arts, 2, 68–85. https://doi.org/10.1037/1931-3896.2.2.68.
So, C. and Joo, J. (2017) Does a persona improve creativity? Des. J., 20, 459–475. https://doi.org/10.1080/14606925.2017.1319672.
Soley, L. C. and Smith, A. L. (2008) Projective techniques for social science and business research. Southshore Press, Milwaukee, Wis.
Stein, M. I. (1947) The use of a sentence completion test for the diagnosis of personality. J. Clin. Psychol., 3, 47–56. https://doi.org/10.1002/1097-4679(194701)3:1<47::AIDJCLP2270030111>3.0.CO;2-Q.
Stephens, M. W. (1970) Stimulus pull as a determinant of individual differences in sentence completion responses. J. Proj Tech. Pers. Assess., 34, 332–339. https://doi.org/10.1080/0091651X.1970.10380258.
Trites, D. K. (1956) Evaluation of assumptions underlying interpretation of sentence completion tests. J. Consult. Psychol., 20, 8. https://doi.org/10.1037/h0044329.
Tsai, W. C. and van den Hoven, E. (2018) Memory probes: exploring retrospective user experience through traces of use on cherished objects. Int. J. Des., 12, 57–72.
Turnbow, K. and Dana, R. H. (1981) The effects of stem length and directions on sentence completion test responses. J. Pers. Assess., 45, 27–32. https://doi.org/10.1207/s15327752jpa4501_7.
Väätäjä, H. K., Savioja, P., Roto, V., Olsson, T. D. and Varsaluoma, J. T. (2015) User Experience Goals as a Guiding Light in Design and Development ? Early Findings. In Beckmann, C. and Gross, T. (eds.), INTERACT 2015 Adjunct Proceedings: 15th IFIP TC.13 International Conference on Human-Computer Interaction. Univ. of Bamberg Press, pp. 521–527.
Van Den Heuvel, R. and Lallemand, C. (2023). Personal Informatics at the Office: User-Driven, Situated Sensor Kits in the Workplace. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work. https://doi.org/10.1145/3596671.3598577.
Verhaegen, P. A., Vandevenne, D., Peeters, J. and Duflou, J. R. (2013) Refinements to the variety metric for idea evaluation. Des. Stud., 34, 243–263. https://doi.org/10.1016/j.destud.2012.08.003.
Vermeeren, A.P.O.S., Law, E.L.C., Roto, V., Obrist, M., Hoonhout, J. and Väänänen-Vainio-Mattila, K. (2010). User Experience Evaluation Methods: Current State and Development Needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI’10). Association for Computing Machinery, New York, NY, USA, pp. 521–530. https://doi.org/10.1145/1868914.1868973
Walsh, T. (2016) Online surveys in collecting cross-cultural qualitative user experience feedback. Tampere University of Technology, Tampere.
Walsh, T., Nurkka, P. and Kujala, S. (2010a) Exploring Cross-Cultural UX Evaluation with the Remote Online Sentence Completion Method Cross-Cultural Design. In Proceedings of the Ninth International Workshop on Internationalisation of Products and Systems IWIPS 2010, 7-10 July 2010. Designing for Global Markets, London, UK, pp. 11–22.
Walsh, T., Nurkka, P. and Walsh, R. (2010b). Cultural differences in smartphone user experience evaluation. In Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia (MUM’10). Association for Computing Machinery, New York, NY, USA, Article 24, pp. 1–9. https://doi.org/10.1145/1899475.1899499
Walsh, T., Nurkka, P., Petrie, H. and Olsson, J. (2013). The Effect of Language in Answering Qualitative Questions in User Experience Evaluation Web-Surveys. In Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration (OzCHI’13). Association for Computing Machinery, New York, NY, USA, pp. 73–82 . https://doi.org/10.1145/2541016.2541049
Weisgerber, C. A. (1969) Psychological Assessment of Candidates for a Religious Order. Loyola University Press, Chicago.