Dark patterns; Deceptive design patterns; Digital vulnerability; EU digital strategy; Risk assessment; Dark pattern; Deceptive design pattern; Design Patterns; Digital architecture; Digital services; Digital strategies; Multi dimensional; Risks assessments; Business, Management and Accounting (all); Computer Networks and Communications; Law
Résumé :
[en] In the last few years, there have been growing concerns about the far-reaching influence that digital architectures may exert on individuals and societies. A specific type of digital manipulation is often engineered into the interfaces of digital services through the use of so-called dark patterns, that cause manifold harms against which nobody seems to be immune. However, many areas of law rely on a traditional class-based view according to which certain groups are inherently more vulnerable than others, such as children. Although the undue influence exerted by dark patterns on online decisions can befall anybody, empirical studies show that there are actually certain factors that aggravate the vulnerability of some people by making them more likely to incur in certain manipulation risks engineered in digital services and less resilient to the related harms. But digital vulnerability does not overlap with traditionally protected groups and depends on multifaceted factors. This article contributes to the ongoing discussions on these topics by offering (i) a multidisciplinary mapping of the micro, meso, and macro factors of vulnerability to dark patterns; (ii) a subsequent critical reflection on the feasibility of the risk assessment proposed in three selected EU legal frameworks: the General Data Protection Regulation, the Digital Services Act, and the Artificial Intelligence Act; (iii) and multidisciplinary suggestions to increase resilience towards manipulative designs online.
Disciplines :
Sciences informatiques
Auteur, co-auteur :
ROSSI, Arianna ; Sant'Anna University of Advanced Studies > LIDER Lab, Dirpolis
Carli, Rachele; AlmaAI, University of Bologna, Italy ; University of Luxembourg, Esch sur Alzette, Luxembourg
Botes, Marietjie W.; Center for Research Evaluation Science and Technology (CREST), Stellenbosch University, Stellenbosch, South Africa ; School of Law, Howard College, University of KwaZulu Natal, Glenwood, South Africa
FERNANDEZ, Angelica ; University of Luxembourg > Faculty of Law, Economics and Finance > Department of Law > Team Mark David COLE
SERGEEVA, Anastasia ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > IRiSC
SANCHEZ CHAMORRO, Lorena ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust > IRiSC > Team Gabriele LENZINI ; University of Twente > Human-Media Interaction Group
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
Who is vulnerable to deceptive design patterns? A transdisciplinary perspective on the multi-dimensional nature of digital vulnerability
Joint Doctorate grant agreement No 814177 LAST-JD-Rights of Internet of Everything PNRR/NextGenerationEU project \u201CBiorobotics Research and Innovation Engineering Facilities \u201CIR0000036\u201D \u2013 CUP J13C22000400007\u201D. L. Dutch Research Council (NWO) via RESOCIAL project (NWA.1540.21.001), from the thematic program \u201CVulnerability and resilience in an online society\u201D
Subventionnement (détails) :
A. Rossi, A. Sergeeva and L. Sanchez Chamorro have received funding from the Luxembourg National Research Fund (FNR)\u2019s grant agreement no. IS/14717072 (DECEPTICON: Deceptive patterns online). A. Fernandez has received funding from the Luxembourg National Research Fund (FNR) (PRIDE17/12251371). R. Carli is supported by the Joint Doctorate grant agreement No 814177 LAST-JD-Rights of Internet of Everything. A. Rossi was also partially funded by the PNRR/NextGenerationEU project \u201CBiorobotics Research and Innovation Engineering Facilities \u201CIR0000036\u201D \u2013 CUP J13C22000400007\u201D. L. S\u00E1nchez Chamorro has also received funding from the Dutch Research Council (NWO) via RESOCIAL project (NWA.1540.21.001), from the thematic program \u201CVulnerability and resilience in an online society\u201D. We thank Sophie Doublet from the University of Luxembourg for her help in designing the graphical abstract. We are grateful to the participants of the Lorentz Center's workshop on \u201CFair design for online interfaces\u201D (held between 29 January and 2 February 2024) for their feedback on the main findings of this research.
Mathur, A., Kshirsagar, M., Mayer, J., What makes a dark pattern. dark? Design attributes, normative considerations, and measurement methods. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, ACM, Yokohama Japan, 1–18, 10.1145/3411764.3445610 URL: https://dl.acm.org/doi/10.1145/3411764.3445610.
OECD. Dark commercial patterns. Number 336 in OECD Digital Economy Papers, Paris, 2022, 10.1787/44f5e846-en URL:.
Chivukula, S.S., Watkins, C., McKay, L., Gray, C.M., “Nothing comes before profit”: asshole design in the wild. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow Scotland UK, 2019, ACM, 1–6, 10.1145/3290607.3312863 URL: https://dl.acm.org/doi/10.1145/3290607.3312863.
Gray, C.M., Kou, Y., Battles, B., Hoggatt, J., Toombs, A.L., The dark (patterns) side of UX design. Proceedings of the 2018 CHI conference on human factors in computing systems, 2018, 1–14.
Sánchez Chamorro, L., Bongard-Blanchy, K., Koenig, V., Ethical tensions in UX design practice: exploring the fine line between persuasion and manipulation in online interfaces. Designing Interactive Systems (DIS), 2023, ACM, Pittsburgh PA USA, 15, 10.1145/3563657.3596013 ACM, NewYork,NY, USApages.
Mathur, A., Acar, G., Friedman, M.J., Lucherini, E., Mayer, J., Chetty, M., Narayanan, A., Dark patterns at scale: findings from a crawl of 11k shopping websites. Proc ACM Hum Comput Interact 3 (2019), 1–32.
Waldman, A.E., Cognitive biases, dark patterns, and the ‘privacy paradox. Curr Opin Psychol 31 (2020), 105–109.
OECD, 2023. Consumer vulnerability in the digital age. Number 355 in OECD Digital Economy Papers, Paris. URL: doi:10.1787/4d013cc5-en.
European Commission, London Economics, VVA Consulting, Ipsos Mori consortium. Consumer Vulnerability Across Key Markets in the European Union. 2016, 10.2818/056024 Final report. LuxembourgURL: https://commission.europa.eu/system/files/2018-04/consumers-approved-report_en.pdf.
Helberger, N., Sax, M., Strycharz, J., Micklitz, H.W., Choice architectures in the digital economy: towards a new understanding of digital vulnerability. J Consum Policy 45 (2022), 175–200, 10.1007/s10603-021-09500-5 URL: https://link.springer.com/10.1007/s10603-021-09500-5.
European Data Protection Board, 2023. Guidelines 03/2022 on deceptive design patterns in social media platform interfaces: how to recognise and avoid them. Version 2.0. URL: https://edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-032022-deceptive-design-patterns-social-media_en.
Competition and Markets Authority, 2022. Online choice architecture - how digital design can harm competition and consumers - discussion paper. URL: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1066524/Online_choice_architecture_discussion_paper.pdf.
Article 29 Data Protection Working Party, 2017. Guidelines on Data Pro- tection Impact Assessment (DPIA) and determining whether processing is “likely to result in a high risk” for the purposes of Regulation 2016/679. URL: https://ec.europa.eu/newsroom/just/document.cfm?doc_id=47711.
Malgieri, G., Niklas, Jędrzej, Vulnerable data subject. Comput Law Security Rev, 37, 2020, 105415, 10.1016/j.clsr.2020.105415 Doi:.
Bosch, C., Erb, B., Kargl, F., Kopp, H., Pfattheicher, S., Tales from the dark side: privacy dark strategies and privacy dark patterns. Proc Priv Enhancing Technol 2016 (2016), 237–254.
Kahneman, D., Thinking, fast and slow. 2011, MacMillian.
Acquisti, A., Sleeper, M., Wang, Y., Wilson, S., Adjerid, I., Balebako, R., Brandimarte, L., Cranor, L.F., Komanduri, S., Leon, P.G., Sadeh, N., Schaub, F., Nudges for privacy and security: understanding and assisting users’ choices online. ACM Comput Surv 50 (2017), 1–41, 10.1145/3054926.
Graßl, P., Schraffenberger, H., Borgesius, F.Z., Buijzen, M., Dark and bright patterns in cookie consent requests. J Dig Soc Res 3 (2021), 1–38.
Utz, C., Degeling, M., Fahl, S., Schaub, F., Holz, T., (un)informed consent: studying GDPR consent notices in the field. Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, 2019, 973–990, 10.1145/3319535.3354212.
Simon, H.A., Rational choice and the structure of the environment. Psychol Rev 63 (1956), 129–138, 10.1037/h0042769 URL : http://doi.apa.org/getdoi.cfm?doi=10.1037/h0042769.
Hodgkinson, G.P., Bown, N.J., Maule, A.J., Glaister, K.W., Pearman, A.D., Breaking the frame: an analysis of strategic cognition and decision making under uncertainty. Strateg Manag J 20 (1999), 977–985.
Nikoli´c, J., Biases in the decision-making process and possibilities of overcoming them. Ekonomski Horizonti, 20, 2018, 45–59.
Hertwig, R., Gru¨ne-Yanoff, T., Nudging and boosting: steering or empowering good decisions. Perspect Psychol Sci 12 (2017), 973–986, 10.1177/1745691617702496.
Kozyreva, A., Lewandowsky, S., Hertwig, R., 2020. Citizens versus the internet: confronting digital challenges with cognitive tools, 54.
Cox, A.L., Gould, S.J., Cecchinato, M.E., Iacovides, I., Renfree, I., Design frictions for mindful interactions: the case for microboundaries. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2016, Association for Computing Machinery, New York, NY, USA, 1389–1397, 10.1145/2851581.2892410 URL:10.1145/2851581.2892410.
Moser, C., Schoenebeck, S.Y., Resnick, P., Impulse buying: design practices and consumer needs. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, ACM Press, Glasgow, Scotland UK, 1–15, 10.1145/3290605.3300472 URL: http://dl.acm.org/citation.cfm?doid=3290605.3300472.
Bongard-Blanchy, K., Rossi, A., Rivas, S., Doublet, S., Koenig, V., Lenzini, G., I Am Definitely Manipulated, Even When I am Aware of IIt. It's Ridiculous!” - dark patterns from the end-user perspective. Designing Interactive Systems Conference 2021, ACM, Virtual Event USA, 2021, 763–776, 10.1145/3461778.3462086 URL: https://dl.acm.org/doi/10.1145/3461778.3462086.
Di Geronimo, L., Braz, L., Fregnan, E., Palomba, F., Bacchelli, A., UI dark patterns and where to find them: a study on mobile applications and user perception. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, 1–14, 10.1145/3313831.3376600 ght/content/10.1108/S2050-206020150000010002/full/html, 10.1108/S2050-206020150000010002.
Matte, C., Bielova, N., Santos, C., Do cookie banners respect my choice?: measuring legal compliance of banners from IAB Europe's tranparency and consent framework. 2020 IEEE Symposium on Security and Privacy (SP), 2020, 791–809, 10.1109/SP40000.2020.00076.
Article 29 Data Protection Working Party, 2018. Guidelines on transparency under regulation 2016/679, 17/en wp260 rev.01. adopted on 29 November 2017. as last revised and adopted on 11 April 2018. URL: https://ec.europa.eu/newsroom/article29/redirection/document/51025.
Nissenbaum, H., A contextual approach to privacy online. Daedalus, 140, 2011, 32–48, 10.1162/DAED_a_00113.
Fineman, M.A., The vulnerable subject: Anchoring equality in the human condition. Yale JL & Feminism, 20, 2008, 1.
Baker, S.M., Gentry, J.W., Rittenburg, T.L., Building understanding of the domain of consumer vulnerability. J Macromarket 25 (2005), 128–139, 10.1177/0276146705280622.
Yeung, K., ‘hypernudge’: big data as a mode of regulation by design. Inf, Commun Soc 20 (2017), 118–136.
European Commission, 2021. Commission Notice – Guidance on the interpretation and application of Directive 2005/29/EC of the European Parliament and of the Council concerning unfair business-to-consumer commercial practices in the internal market. URL: https://op.europa.eu/en/publication-detail/-/publication/c608fff7-687a-11ec-9136-01aa75ed71a1/language-en.
ISO, 2023. ISO 31700-1:2023 consumer protection — privacy by design for consumer goods and services — part 1: high-level requirements.
European Commission. Behavioural study on unfair commercial practices in the digital environment: dark patterns and manipulative personalisation: final report. 2022, Publications Office of the European Union, LU, 10.2838/859030 URL: data.eur opa.eu/doi/.
Gunawan, J., Santos, C., Kamara, I., Redress for dark patterns privacy harms? a case study on consent interactions. Proceedings of the 2022 Symposium on Computer Science and Law, 2022, 181–194.
European Commission, 2022. Digital fairness – fitness check on EU consumer law. URL: https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/13413-Digital-fairness-fitness-check-on-EU-consumer-law_en.
ICO, 2022. Overview of data protection harms and the ICO taxonomy v1. URL: https://ico.org.uk/media/about-the-ico/douments/4020144/overview-of-data-protection-harms-and-the-ico-taxonomy-v1-202204.pdf.
Courmont, A., 2022. Le plaignant type? Un homme, diplômé et cadre. URL: https://linc.cnil.fr/fr/le-plaignant-type-un-homme-diplome-et-cadre.
PenzeyMoog, E., 2021. Design for safety. a book apart.
Kaplan, S., Garrick, B.J., On the quantitative definition of risk. Risk Anal 1 (1981), 11–27.
Efroni, Z., The Digital Services Act: risk-based regulation of online platforms. Internet Policy Rev, 2021 URL: https://policyreview.info/articles/news/digital-services-act-risk-based-regulation-online-platforms/1606.
Chivukula, S.S., Gray, C.M., Brier, J.A., Analyzing value discovery in design decisions through ethicography. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland UK, 2019, 1–12, 10.1145/3290605.3300307 URL: https://dl.acm.org/doi/10.1145/3290605.3300307.
Chivukula, S.S., Brier, J., Gray, C.M., Dark intentions or persuasion? UX designers’ activation of stakeholder and user values. Proceedings of the 2018 ACM Conference Companion Publication on Designing Interative Systems, 2018, ACM, Hong Kong China, 87–91, 10.1145/3197391.3205417 URL: https://dl.acm.org/doi/10.1145/3197391.3205417.
Norman, D.A., Emotional design: why we love (or hate) everyday things. 2004, Civitas Books.
Burgemeestre, B., Hulstijn, J., Tan, Y.H., Rule-based versus principle- based regulatory compliance. 2009, IOS Press, 37–46.
Calo, R., Privacy, vulnerability, and affordance. DePaul L Rev, 66, 2016, 591.
European Data Protection Board, 2019. Guidelines 4/2019 on article 25 data protection by design and by default. URL: https://edpb.europa.eu/sites/edpb/files/consultation/edpb_guidelines_201904_dataprotection_by_design_and_by_default.pdf.
Susser, D., Roessler, B., Nissenbaum, H.F., Online manipulation: Hid- den influences in a digital world. Geo L Tech Rev, 4, 2018, 1.
Article 29 Data Protection Working Party, 2018. Guidelines on Automated individual decision- making and Profiling for the purposes of Regulation 2016/679 17/EN WP251rev.01.
Luna, F., Elucidating the concept of vulnerability: layers not labels. IJFAB: Int J Femin Approaches Bioethics 2 (2009), 121–139.
Leiser, M.R., Santos, C., Dark patterns, enforcement, and the emerging digital design acquis: manipulation beneath the interface. enforcement, and the emerging digital design acquis: manipulation beneath the interface. Eur J Law Technol, 15(1), 2024 https://ejlt.org/index.php/ejlt/article/view/990.
Mantelero, A., 2023. Fundamental rights impact assessment in the DSA. Verfassungsblog ed. Berlin. p. 107–119.
Seymour, W., Van Kleek, M., Exploring interactions between trust, anthropomorphism, and relationship development in voice assistants. Proceedings of the ACM on Human-Computer Interaction 5, 371, 2021, 1–371, 10.1145/3479515 16URL:10.1145/3479515.
Lacey, C., Caudwell, C., Cuteness as a ‘dark pattern’ in home robots. 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2019, IEEE, 374–381.
De Conca, S., The present looks nothing like the Jetsons - deceptive design in virtual assistants and the protection of the rights of users. Comput Law Security Rev, 51, 2023, 105866, 10.1016/j.clsr.2023.105866 URL:.
Carli, R., Calvaresi, D., Reinterpreting vulnerability to tackle deception in principles-based XAI for human-computer interaction. International workshop on explainable, transparent autonomous agents and multi- agent systems, 2023, Springer, 249–269.
Natale, S., Deceitful media: artificial intelligence and social life after the Turing test. 2021, Oxford University Press, USA.
Bermudez, J.P., Nyrup, R., Deterding, S., Moradbakhti, L., Mougenot, C., You, F., Calvo, R.A., What is a subliminal technique? An ethical perspective on ai-driven influence. 2023 IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS), 2023, IEEE, West Lafayette, IN, USA, 1–10, 10.1109/ETHICS57328.2023.10155039 URL: https://ieeexplore.ieee.org/document/10155039/.
Laux, J., Wachter, S., Mittelstadt, B., Trustworthy artificial intelligence and the European union AI Act: on the conflation of trustworthiness and acceptability of risk. Regul Gov, 2023.
Carli, R., Najjar, A., Calvaresi, D., Risk and exposure of XAI in persuasion and argumentation: the case of manipulation. International workshop on explainable, transparent autonomous agents and multi- agent systems, 2022, Springer, 204–220.
Wäfler, T., Schmid, U., Explainability is not enough: requirements for human-AI-partnership in complex socio-technical systems. Proceedings of the 2nd European Conference on the Impact of Artificial Intelligence and Robotics (ECIAIR 2020), 2021, 184–194, 10.20378/irb-49775.
Haataja, M., Bryson, J.J., The European Parliament's AI regulation: should we call it progress? Series 2, 4, 2022, Amicus Curiae, 707.
Ebers, M., Hoch, V.R., Rosenkranz, F., Ruschemeier, H., Steinrotter, B., 2021. The European Commission's proposal for an Artificial Intelligence Act—a critical assessment by members of the robotics and ai law society (rails). J 4, 589–603.
Smuha, N.A., Ahmed-Rengers, E., Harkens, A., Li, W., MacLaren, J., Piselli, R., Yeung, K., 2021. How the EU can achieve legally trustworthy AI: a response to the European Commission's proposal for an Artificial Intelligence Act. URL: https://ssrn.com/abstract=3899991 or doi:10.2139/ssrn.3899991.
Stahl, B.C., Rodrigues, R., Santiago, N., Macnish, K., A European agency for artificial intelligence: protecting fundamental rights and ethical values. Comput Law Security Rev, 45, 2022, 105661.
Zanca, F., Brusasco, C., Pesapane, F., Kwade, Z., Beckers, R., Avanzo, M., Regulatory aspects of the use of artificial intelligence medical software. Seminars in radiation oncology, 2022, Elsevier, 432–441.
Gibson, J.J., The theory of affordances. 1986, Lawrence Erlbaum Associates, 127–137.
Helsper, E.J., Digital world: from divides to socio-digital inequalities. The digital disconnect. The social causes and consequences of digital inequalities, 2021, SAGE Publications, 27–45.
Norman, D.A., The design of everyday things. Revised and expanded edition ed., 2013, Basic Books, New York, New York.
Bronfenbrenner, U., The ecology of human development: experiments by nature and design. 1979, Harvard University Press.
Bronfenbrenner, U., Morris, P.A., The bioecological model of human development. Handbook of child psychology, 2007, Wiley, 793–828.
Johnson, G.M., Puplampu, K.P., Internet use during childhood and the ecological techno-subsystem. Canadian J Learn Technol /La revue canadienne de l'apprentissage et de la technologie, 34, 2008, 10.21432/T2CP4T URL: http://www.cjlt.ca/index.php/cjlt/article/view/26428.
O'Neill, B., Ecological perspectives and children's use of the internet: exploring micro to macro level analysis. Eesti Haridusteaduste Ajakiri Estonian J Educ 3 (2015), 32–53, 10.12697/eha.2015.3.2.02b.
Murnane, E.L., Walker, T.G., Tench, B., Voida, S., Snyder, J., Personal informatics in interpersonal contexts: towards the design of technology that supports the social ecologies of long-term mental health management. Proc ACM Hum Comput Interact 2:CSCW (2018), 1–27, 10.1145/3274396.
Costa Figueiredo, M., Chen, Y, Health data in fertility care: an ecological perspective. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, 1–17, 10.1145/3411764.3445189.
Malgieri, G., Who is the vulnerable individual?. Vulnerability and Data Protection Law, 1st edition, 2023, Oxford University Press, Oxford https://academic.oup.com/book/46055/chapter/404533553.
Sánchez Chamorro, L., Lallemand, C., Gray, Colin, M., My mother told me these things are always fake” — understanding teenagers’ experiences with manipulative designs. Designing Interactive Systems Conference (DIS ’24), July 1–5, 2024, 2024, IT University of Copenhagen, Denmark, 13, 10.1145/3643834.3660704 ACM, New York, NY, USApages.
Mantelero, A., The future of consumer data protection in the E.U. Rethinking the “notice and consent” paradigm in the new era of predictive analytics. Comput Law Security Rev 30 (2014), 643–660, 10.1016/j.clsr.2014.09.004 URL: https://linkinghub.elsevier.com/retrieve/pii/S026736491400154X.
Lessig, Lawrence, Code and other laws of cyberspace. 1999, Basic Books, Inc.Division of HarperCollins 10 E. 53rd St, New York, NY United States.
Lockton, D., Persuasive technology and digital design for behaviour change. SSRN Electron J, 2012, 10.2139/ssrn.2125957 URL http://www.ssrn.com/abstract=2125957.
Botes, M., Autonomy and the social dilemma of online manipulative behavior. AI Ethics, 2022, 10.1007/s43681-022-00157-5 URL: link.springer.com/10.1007/s43681-022-00157-5.
Gray, C.M., Chivukula, S.S., Ethical mediation in UX practice. Proceedings of the 2019 CHI conference on human factors in computing systems, ACM, Glasgow Scotland Uk, 2019, 1–11, 10.1145/3290605.3300408 URL: https://dl.acm.org/doi/10.1145/3290605.3300408.
Wong, R.Y., Tactics of soft resistance in user experience professionals’ values work. Proc ACM Hum Comput Interact 5 (2021), 1–28, 10.1145/3479499.
Constanza-Chock, S., Design values: hard-coding liberation? In design justice. Community-led practices to build the worlds we need, 2020, The MIT Press, 48–89.
Käufer, S., Chemero, A., Phenomenology: an introduction. 2021, Wiley URL: https://books.google.lu/books?id=qR_azQEACAAJ.
Hicks, J.M., Still, J.D., Examining the effects of clutter and target salience in an e-commerce visual search task. Proceedings of the human factors and ergonomics society 2019 annual meeting, 2019, 1761–1765.
Armel, K.C., Beaumel, A., Rangel, A., Biasing simple choices by manipulating relative visual attention. Judgm Decis Mak 3 (2008), 396–403.
Still, J., Still, M., Influence of visual salience on webpage product searches. ACM Trans Appl Percept (TAP) 16 (2019), 1–11.
Milosavljevic, M., Navalpakkam, V., Koch, C., Rangel, A., Relative visual saliency differences induce sizable bias in consumer choice. J Consum Psychol 22 (2012), 67–74.
Towal, R.B., Milosavljevic, M., Koch, C., The effect of visual salience on multiple-alternative, value-based decisions. J Vis, 12, 2012, 167 –167.
Veas, E.E., Mendez, E., Feiner, S.K., Schmalstieg, D., Directing attention and influencing memory with visual saliency modulation. Proceedings of the SIGCHI conference on human factors in computing systems, 2011, 1471–1480.
Jarovsky, L., Dark patterns in personal data collection: definition, taxonomy and lawfulness. SSRN Electron J, 2022, 048582, 10.2139/ssrn.4048582 URL https://papers.ssrn.com/abstract=4.
Gray, C.M., Sanchez Chamorro, L., Obi, I., Duane, J.N., Mapping the landscape of dark patterns scholarship: a systematic literature review. Designing interactive systems conference (DIS companion ’23), 2023, 10.1145/3563703.3596635.
Nouwens, M., Liccardi, I., Veale, M., Karger, D., Kagal, L., Dark patterns after the GDPR: scraping consent pop-ups and demonstrating their influence. CHI Conference on Human Factors in Computing Systems, 2020, 13, 10.1145/3313831.3376321 URL: http://arxiv.org/abs/2001.02479 arXiv: 2001.02479.
Luguri, J., Strahilevitz, L.J., Shining a light on dark patterns. J Legal Anal 13 (2021), 43–109, 10.1093/jla/laaa006.
Berens, B.M., Dietmann, H., Krisam, C., Kulyk, O., Volkamer, M., Cookie disclaimers: impact of design and users’ attitude. Proceedings of the 17th International Conference on Availability, Reliability and Security, 2022, ACM, Vienna Austria, 1–20, 10.1145/3538969.3539008 URL: dl.acm.org/doi/10.1145/3538969.3539008.
Gray, C.M., Chen, J., Chivukula, S.S., Qu, L., End user accounts of dark patterns as felt manipulation. Proceedings of the ACM in human-computer interaction, 5, 2021, 26, 10.1145/3479516 URL:.
Maier, M., Harr, R., Dark design patterns: an end-user perspective. Hum Technol 16 (2020), 170–199, 10.17011/ht/urn.202008245641.
Borberg, I., Hougaard, R., Rafnsson, W., Kulyk, O., So I Sold My Soul”: effects of dark patterns in cookie notices on end-user behavior and perceptions. Workshop on Usable Security and Privacy (USEC), 2022, 11.
Helsper, E.J., Survey on the use of information and communication technologies in Brazilian households: ICT households 2015. 2016, Núcleo de Informação e Coordenação do Ponto BR, São Paulo, Brasil.
Scheerder, A., van Deursen, A., van Dijk, J., Taking advantage of the internet: a qualitative analysis to explain why educational background is decisive in gaining positive outcomes. Poetics, 80, 2020, 101426, 10.1016/j.poetic.2019.101426.
Van Deursen, A.J.A.M., Helsper, E.J., The third-level digital divide: who benefits most from being online?, 10, 2015, Emerald Group Publishing Limited, 29–52 URL: https://www.emerald.com/insi.
Van Dijk, J., van Deursen, A.J.A.M., Digital skills. 2014, Palgrave Macmillan US, New York, 10.1057/9781137437037 URL: http://link.springer.com/10.1057/9781137437037.
Eynon, R., Geniets, A., The digital skills paradox: how do digitally excluded youth develop skills to use the internet?. Learn Media Technol 41 (2016), 463–479, 10.1080/17439884.2014.1002845.
Freese, J., Rivas, S., Hargittai, E., Cognitive ability and internet use among older adults. Poetics 34 (2006), 236–249, 10.1016/j.poetic.2006. 05.008.
Hargittai, E., Hinnant, A., Digital inequality: Differences in young adults’ use of the internet. Commun Res 35 (2008), 602–621, 10.1177/0093650208321782.
Helsper, E.J., van Deursen, A.J.A.M., Do the rich get digitally richer? quantity and quality of support for digital engagement. Inf, Commun Soc 20 (2017), 700–714, 10.1080/1369118X.2016.1203454.
Van Dijk, J., Hacker, K., The digital divide as a complex and dynamic phenomenon. Inf Soc 19 (2003), 315–326, 10.1080/01972240309487.
Zac, A., Huang, Y.-C., von Moltke, A., Decker, C., Ezrachi, A. Dark patterns and consumer vulnerability (August 22, 2023). Available at SSRN: https://ssrn.com/abstract=4547964.
DiPaola, D., Calo, R., Socio-digital vulnerability (SSRN Scholarly Paper 4686874). doi:10.2139/ssrn.4686874.
Zingales, L., Morton, F.S., Rolnik, G., 2019. Stigler committee on digital plat- forms: final report. URL: https://www.chicagobooth.edu/-/media/research/stigler/pdfs/digital-platforms–committee-report–stigler-center.pdf.
Tsetsi, E., Rains, S.A., Smartphone internet access and use: extending the digital divide and usage gap. Mob Media Commun 5 (2017), 239–255, 10.1177/2050157917708329.
Gunawan, J., Pradeep, A., Choffnes, D., Hartzog, W., Wilson, C., A comparative study of dark patterns across web and mobile modalities. Proc ACM Hum Comput Interact 5 (2021), 1–29, 10.1145/3479521.
Lee, C.S., McKenzie, K., Socioeconomic and geographic inequalities of internet addiction in Korean adolescents. Psychiatry Investig 12 (2015), 559–562, 10.4306/pi.2015.12.4.559.
Radesky, J., Hiniker, A., McLaren, C., Akgun, E., Schaller, A., Weeks, H.M., Campbell, S., Gearhardt, A.N., Prevalence and characteristics of manipulative design in mobile applications used by children. JAMA Netw Open, 5, 2022, e2217641 –e2217641.
Kampanos, G., Shahandashti, S.F., Accept all: the landscape of cookie banners in Greece and the UK. Jøsang, A., Futcher, L., Hagen, J., (eds.) ICT systems security and privacy protection, 2021, Springer International Publishing, Cham, 213–227, 10.1007/978-3-030-78120-0_14.
Santos, C., Rossi, A., Sanchez Chamorro, L., Bongard-Blanchy, K., Abu- Salma, R., Cookie banners, what's the purpose? Analyzing cookie banner text through a legal lens. Proceedings of the 20th Workshop on Workshop on Privacy in the Electronic Society, 2021, ACM, 187–194, 10.1145/3463676.3485611 Virtual Event Republic of KoreaURL: https://dl.acm.org/doi/10.1145/3463676.3485611.
Bermeitinger, C., Priming. Psychology and mental health: concepts, methodologies, tools, and applications, 2016, IGI Global, 42–88.
Hsu, N., Schu¨tt, Z., Psychology of priming. 2012, Nova Science Publishers.
Breitmeyer, B.G., Ro, T., Singhal, N.S., Unconscious color priming occurs at stimulus-not percept-dependent levels of processing. Psychol Sci 15 (2004), 198–202.
Chartrand, T.L., van Baaren, R.B., Bargh, J.A., Linking automatic evaluation to mood and information processing style: consequences for experienced affect, impression formation, and stereotyping. J Exp Psychol: General, 135, 2006, 70.
Francken, J.C., van Gaal, S., de Lange, F.P., Immediate and long-term priming effects are independent of prime awareness. Conscious Cogn 20 (2011), 1793–1800.
Yeun Chun, K., Hee Song, J., Hollenbeck, C.R., Lee, J.H., Are contextual advertisements effective? the moderating role of complexity in banner advertising. Int J Advert 33 (2014), 351–371.
Yi, Y., The influence of contextual priming on advertising effects. Adv Consume Res, 18, 1991, 1.
Yoo, C.Y., Unconscious processing of web advertising: effects on implicit memory, attitude toward the brand, and consideration set. J Interact Market 22 (2008), 2–18.
Wänke, M., Primes as hidden persuaders. Curr Opin Psychol 12 (2016), 63–66.
Petticrew, M., Maani, N., Pettigrew, L., Rutter, H., Van Schalkwyk, M.C., Dark nudges and sludge in big alcohol: behavioral economics, cognitive biases, and alcohol industry corporate social responsibility. Milbank Q 98 (2020), 1290–1328.
Costello, F.J., Yun, J., Lee, K.C., Digital dark nudge: an exploration of when digital nudges unethically depart. Proceedings of the 55th Hawaii International Conference on System Sciences, 2022.
Dennis, A.R., Yuan, L., Feng, X., Webb, E., Hsieh, C.J., Digital nudging: numeric and semantic priming in e-commerce. J Manage Inf Syst 37 (2020), 39–65.
Jiang, X., Jiang, Y., Parasuraman, R., The visual priming of motion- defined 3d objects. PLoS One, 10, 2015, e0144730.
Kareklas, I., Muehling, D.D., King, S., The effect of color and self-view priming in persuasive communications. J Bus Res 98 (2019), 33–49.
Elgendi, M., Kumar, P., Barbic, S., Howard, N., Abbott, D., Cichocki, A., Subliminal priming—state-of-the-art and future perspectives. Behav Sci, 8, 2018, 54.
Matthews, G., Hancock, P.A., The handbook of operator fatigue. 2017, CRC Press.
Crump, C., Walenchok, S., Johnson, C., Pauszek, J., Young, D., Stress, operator error, and transportation accidents. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2021, SAGE Publications Sage CA, Los Angeles, CA, 1453–1458.
Hockey, G.R.J., Operator functional state: the assessment and pre- diction of human performance degradation in complex tasks, 355, 2003, IOS Press.
Kahol, K., Leyba, M.J., Deka, M., Deka, V., Mayes, S., Smith, M., Fer- rara, J.J., Panchanathan, S., Effect of fatigue on psychomotor and cognitive skills. Am J Surg 195 (2008), 195–204.
Ricciardi, O., Maggi, P., Nocera, F.D., Boredom makes me nervous: fidgeting as a strategy for contrasting the lack of variety. Int J Hum Factors Ergon 6 (2019), 195–207.
Chaudhary, A., Saroha, J., Monteiro, K., Forbes, A.G., Parnami, A., Are You Still Qatching?”: exploring unintended user behaviors and dark patterns on video streaming platforms. Designing Interactive Systems Conference, ACM, Virtual Event Australia, 2022, 776–791, 10.1145/3532106.3533562 URL: https://dl.acm.org/doi/10.1145/3532106.3533562.
Zagal, J.P., Bj¨ork, S., Lewis, C., Dark patterns in the design of games. Foundations of digital games, 2013, 2013.
Amirpur, M., Benlian, A., Buying under pressure: purchase pressure cues and their effects on online buying decisions. Proceedings of the Thirty Sixth International Conference on Information Systems, 2015, 1–18.
Bielova, N., Litvine, L., Nguyen, A., Chammat, M., Toubiana, V., & Hary, E. (2024). The effect of design patterns on (Present and Future) cookie consent decisions. Proceedings of the 33rd USENIX Security Symposium, 1–9.
Avolicino, S., Di Gregorio, M., Palomba, F., Romano, M., Sebillo, M., & Vitiello, G. (2022). AI-based emotion recognition to study users’ perception of dark patterns. In M. Kurosu, S. Yamamoto, H. Mori, M. M. Soares, E. Rosenzweig, A. Marcus, P.-L. P. Rau, D. Harris, & W.-C. Li (Eds.), HCI International 2022—Late Breaking Papers. Design, User Experience and Interaction (Vol. 13516, pp. 185–203). Springer International Publishing. doi:10.1007/978-3-031-17615-9_13.
van Nimwegen, C., & de Wit, J. (2022). Shopping in the dark: effects of platform choice on dark pattern recognition. In M. Kurosu (Ed.), Human-computer interaction. User experience and behavior (Vol. 13304, pp. 462–475). Springer International Publishing. doi:10.1007/978-3-031-05412-9_32.
Sánchez Chamorro, L., Toebosch, R., Lallemand, C., Manipulative design and older adults: co-creating magic machines to understand experiences of online manipulation. Designing Interactive Systems Conference (DIS ’24), July 1–5, 2024, 2024, IT University of Copenhagen, Denmark, 17, 10.1145/3643834.3661513 ACM, New York, NY, USApages.
Meyer, M., Adkins, V., Yuan, N., Weeks, H.M., Chang, Y.J., Radesky, J., Advertising in young children's apps: a content analysis. J Dev Behav Pediatr 40 (2019), 32–39.
Forbrukerradet, 2022. Insert coin. How the gaming industry exploits consumers using loot boxes. URL: https://storage.forbrukerradet.no/media/2022/05/2022-05-31-insert-coin-publish.pdf.
Clarke, J.M., Mehrnezhad, M., Toreini, E., 2023. Invisible, unreadable, and inaudible cookie notices: an evaluation of cookie notices for users with visual impairments URL: http://arxiv.org/abs/2308.11643, 10.48550/arXiv.2308.11643. arXiv:2308.11643 [cs].
Sannon, S., Forte, A., Privacy research with marginalized groups: what we know, what's needed, and what's next. Proceedings of the ACM on Human-Computer Interaction, 6, 2022, 1–33, 10.1145/3555556.
Holkar, M., Lees, C., A safer bet? Online gambling and mental health. 2020 London, UK. URL https://www.moneyandmentalhealth.org/wp-content/uploads/2020/07/A_Safer_Bet.pdf.pdf.
Xia, B., Lu, Q., Perera, H., Zhu, L., Xing, Z., Liu, Y., Whittle, J., Towards concrete and connected AI risk assessment (c2aira): a systematic mapping study. 2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN), 2023, 104–116, 10.1109/CAIN58948.2023.00027.
Beimborn, M., Kadi, S., Köberer, N., Mühleck, M., Spindler, M., Focusing on the human: interdisciplinary. Aging and Technology, 2016, Transcript Verlag.
O'Brien, J.E., Brewer, K.B., Jones, L.M., Corkhum, J., Rizo, C.F., Rigor and respect: recruitment strategies for engaging vulnerable populations in research. J Interpers Violence 37 (2022), NP17052–NP17072, 10.1177/08862605211023497.
Slupska, J., Tanczer, L.M., Threat modeling intimate partner violence: tech abuse as a cybersecurity challenge in the Internet of Things. 2021, Emerald Publishing Limited, 663–688, 10.1108/978-1-83982-848-520211049 Emerald Studies In Digital Crime, Technol- ogy and Social HarmsURL: doi.org/10.1108/97 8-1-83982-848-520211049.
Siciliani, P., Riefa, C., Gamper, H., Consumer theories of harm: an economic approach to consumer law enforcement and policy making. 2019, Bloomsbury Publishing.
BEUC, 2022. “Dark patterns” and the EU consumer law. Recommendations for better enforcement and reform. URL: https://www.beuc.eu/sites/default/files/publications/beuc-x-2022-013_dark_patters_paper.pdf.
Hasan Mansur, S.M., Salma, S., Awofisayo, D., Moran, K., Aidui: toward automated recognition of dark patterns in user interfaces. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), 2023, 1958–1970, 10.1109/ICSE48619.2023.00166.
Kirkman, D., Vaniea, K., Woods, D.W., Darkdialogs: automated detection of 10 dark patterns on cookie dialogs. 8th IEEE European Symposium on Security and Privacy, 2023, IEEE.
Rossi, A., Ducato, R., Haapio, H., Passera, S., When design met law: design patterns for information transparency. Droit de la Consommation= Consumenterecht: DCCR, 2019, 79–121.
Holmes, K., The cycle of exclusion. 2020, MIT Press, 27–40.
Mendez, C., Letaw, L., Burnett, M., Stumpf, S., Sarma, A., Hilderbrand, C., From gendermag to inclusivemag: an inclusive design meta-method. 2019 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC), 2019, IEEE, Memphis, TN, USA, 97–106, 10.1109/VLHCC.2019.8818889 URL: https://ieeexplore.ieee.org/document/8818889/.
Chivukula, S.S., Watkins, C.R., Manocha, R., Chen, J., Gray, C.M., Dimensions of ux practice that shape ethical awareness. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, ACM, Honolulu HI, USA, 1–13, 10.1145/3313831.3376459 URL: https://dl.acm.org/doi/10.1145/3313831.3376459.
