The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings

Yilma, Bereket Abera; Leiva, Luis A.

doi:10.1145/3544548.3581477

Download

Paper published in a book (Scientific congresses, symposiums and conference proceedings)

The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings

Yilma, Bereket Abera; Leiva, Luis A.

2023 • In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)

Peer reviewed

Permalink
https://hdl.handle.net/10993/54495

DOI
10.1145/3544548.3581477

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

chi23-783_access.pdf

Author preprint (7.71 MB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Recommendation systems; Personalization; Machine Learning

Abstract :

[en] Artwork recommendation is challenging because it requires understanding how users interact with highly subjective content, the complexity of the concepts embedded within the artwork, and the emotional and cognitive reflections they may trigger in users. In this paper, we focus on efficiently capturing the elements (i.e., latent semantic relationships) of visual art for personalized recommendation. We propose and study recommender systems based on textual and visual feature learning techniques, as well as their combinations. We then perform a small-scale and a large-scale user-centric evaluation of the quality of the recommendations. Our results indicate that textual features compare favourably with visual ones, whereas a fusion of both captures the most suitable hidden semantic relationships for artwork recommendation. Ultimately, this paper contributes to our understanding of how to deliver content that suitably matches the user's interests and how they are perceived.

Disciplines :

Computer science

Author, co-author :

Yilma, Bereket Abera ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

Leiva, Luis A. ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

yes

Language :

English

Title :

The Elements of Visual Art Recommendation: Learning Latent Semantic Representations of Paintings

Publication date :

April 2023

Event name :

Conference on Human Factors in Computing Systems (CHI ’23)

Event date :

22-04-2023

Audience :

International

Main work title :

Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23)

Peer reviewed :

Peer reviewed

Focus Area :

Computational Sciences

European Projects :

HE - 101071147 - SYMBIOTIK - Context-aware adaptive visualizations for critical decision making

FnR Project :

FNR15722813 - Brainsourcing For Affective Attention Estimation, 2021 (01/02/2022-31/01/2025) - Luis Leiva

Funders :

CE - Commission Européenne [BE]

Available on ORBilu :

since 01 March 2023

Statistics

Number of views

293 (57 by Unilu)

Number of downloads

108 (18 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

Bibliography

Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. 2001. On the Surprising Behavior of Distance Metrics in High Dimensional Space. In Database Theory-ICDT 2001, Jan Van den Bussche and Victor Vianu (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 420-434
Maha Amami, Gabriella Pasi, Fabio Stella, and Rim Faiz. 2016. An lda-based approach to scientifc paper recommendation. In International conference on applications of natural language to information systems. Springer, 200-210
Rel Guzman Apaza, Elizabeth Vera Cervantes, Laura Cruz Quispe, and José Ochoa Luna. 2014. Online Courses Recommendation based on LDA.. In SIMBig. Citeseer, 42-48
LM Aroyo, Y Wang, R Brussee, Peter Gorgels, LW Rutledge, and N Stash. 2007. Personalized museum experience: The Rijksmuseum use case. In Museums and the Web 2007 (San Francisco CA, USA, April 11-14, 2007. Proceedings). Archives & Museum Informatics
Monika Bansal, Munish Kumar, Manish Kumar, and Krishan Kumar. 2021. An efcient technique for object recognition using Shi-Tomasi corner detection algorithm. Soft Computing 25, 6 (2021), 4423-4432
Catarina Barata, M Emre Celebi, and Jorge S Marques. 2018. A survey of feature extraction in dermoscopy image analysis of skin cancer. IEEE journal of biomedical and health informatics 23, 3 (2018), 1096-1109
Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence 35, 8 (2013), 1798-1828
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. 1999. When Is "Nearest Neighbor" Meaningful?. In Database Theory-ICDT'99, Catriel Beeri and Peter Buneman (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 217-235
DavidMBlei, AndrewY Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993-1022
Fidel Cacheda, Víctor Carneiro, Diego Fernández, and Vreixo Formoso. 2011. Comparison of collaborative fltering algorithms: Limitations of current tech-niques and proposals for scalable, high-performance recommender systems. ACM Transactions on the Web (TWEB) 5, 1 (2011), 1-33
Michael Calonder, Vincent Lepetit, Christoph Strecha, and Pascal Fua. 2010. Brief: Binary robust independent elementary features. In European conference on computer vision. Springer, 778-792
Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacifc-Asia conference on knowledge discovery and data mining. Springer, 160-172
Jie Chen, Li-hui Zou, Juan Zhang, and Li-hua Dou. 2009. The Comparison and Application of Corner Detection Algorithms. Journal of multimedia 4, 6 (2009
Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Boston, MA, USA) (SIGIR '09). Association for Computing Machinery, New York, NY, USA, 758-759. https://doi.org/10.1145/1571941.1572114
Dan Cosley, Shyong K Lam, Istvan Albert, Joseph A Konstan, and John Riedl. 2003. Is seeing believing? How recommender system interfaces afect users' opinions. In Proceedings of the SIGCHI conference on Human factors in computing systems. 585-592
Louis Deladiennee and Yannick Naudet. 2017. A graph-based semantic recom-mender system for a refective and personalised museum visit: Extended abstract. In 2017 12th International Workshop on Semantic and Social Media Adaptation and Personalization (SMAP). 88-89. https://doi.org/10.1109/SMAP.2017.8022674
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018
Roman Egger and Joanne Yu. 2021. Identifying hidden semantic structures in Instagram data: a topic modelling comparison. Tourism Review (2021
Abigail R Esman. 2012. TheWorld's Strongest Economy? The Global Art Market
John H Falk. 2016. Identity and the museum visitor experience. Routledge
Angus Graeme Forbes, Saiph Savage, and Tobias Höllerer. 2012. Visualizing and verifying directed social queries. In IEEE Workshop on Interactive Visual Text Analytics. Seattle, WA, Citeseer. Citeseer
Sarah Frost, Manu Mathew Thomas, and Angus G Forbes. 2019. Art i don't like: An anti-recommender system for visual art. In Proceedings of Museums and the Web
Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022
Willem Robert van Hage, Natalia Stash, Yiwen Wang, and Lora Aroyo. 2010. Finding your way through the Rijksmuseum with an adaptive mobile museum guide. In Extended semantic web conference. Springer, 46-59
Hebatallah A Mohamed Hassan, Giuseppe Sansonetti, Fabio Gasparetti, Alessan-dro Micarelli, and Joeran Beel. 2019. Bert, elmo, use and infersent sentence encoders: The panacea for research-paper recommendation?. In RecSys (Late-Breaking Results). 6-10
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770-778
Ruining He, Chen Fang, Zhaowen Wang, and Julian McAuley. 2016. Vista: a visually, socially, and temporally-aware model for artistic recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. 309-316
Ruining He and Julian McAuley. 2016. VBPR: visual bayesian personalized ranking from implicit feedback. In Proceedings of the AAAI conference on artifcial intelligence, Vol. 30
D Helal, H Maxson, and J Ancelet. 2013. Lessons learned: Evaluating the Whitney's multimedia guide. línea)., acceso 25 (2013).http://mw2013.museumsandtheweb.com/paper/lessons-learned-evaluating-the-whitneys-multimedia-guide
María Hernández-Rubio, Alejandro Bellogín, and Iván Cantador. 2020. Aspect-based active learning for user preference elicitation in recommender systems. In Proc. CIRCLE Workshop
A Victor Ikechukwu, S Murali, R Deepu, and RC Shivamurthy. 2021. ResNet-50 vs VGG-19 vs training from scratch: a comparative analysis of the segmentation and classifcation of Pneumonia from chest X-ray images. Global Transitions Proceedings 2, 2 (2021), 375-381
Hamed Jelodar, Yongli Wang, Chi Yuan, Xia Feng, Xiahui Jiang, Yanchao Li, and Liang Zhao. 2019. Latent Dirichlet Allocation (LDA) and Topic modeling: models, applications, a survey. Multimedia Tools and Applications 78, 11 (2019), 15169-15211
Budi Juarto and Abba Suganda Girsang. 2021. Neural Collaborative with Sentence BERT for News Recommender System. JOIV: International Journal on Informatics Visualization 5, 4 (2021), 448-455
Marius Kaminskas and Derek Bridge. 2016. Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems. ACM Trans. Interact. Intell. Syst. 7, 1 2016
Mehmet Oguz Kelek, Nurullah Calik, and Tulay Yildirim. 2019. Painter classif-cation over the novel art painting data set via the latest deep neural networks. Procedia Computer Science 154 (2019), 369-376
Kalliopi Kontiza, Olga Loboda, Louis Deladiennee, Sylvain Castagnos, and Yan-nick Naudet. 2018. A museum app to trigger users' refection. In International Workshop on Mobile Access to Cultural Heritage (MobileCH2018
Alex Krizhevsky, Ilya Sutskever, and Geofrey E Hinton. 2017. Imagenet classi-fcation with deep convolutional neural networks. Commun. ACM 60, 6 (2017), 84-90
Tsvi Kufik, Zvi Boger, and Massimo Zancanaro. 2012. Analysis and Prediction of Museum Visitors' Behavioral Pattern Types. 161-176
Tsvi Kufik, Einat Minkov, and Keren Kahanov. 2014. Graph-based Recommenda-tion in the Museum. In DMRS. Citeseer, 46-48
Dor Lavi, Volodymyr Medentsiy, and David Graus. 2021. consultantbert: Fine-tuned siamese sentence-bert for matching jobs and job seekers. arXiv preprint arXiv:2109.06501 2021
Bin Li and Dimas Lima. 2021. Facial expression recognition via ResNet-50. International Journal of Cognitive Computing in Engineering 2 (2021), 57-64
Peng Liu, Lemei Zhang, and Jon Atle Gulla. 2020. Dynamic attention-based ex-plainable recommendation with textual and visual fusion. Information Processing & Management 57, 6 (2020), 102099
Ioanna Lykourentzou, Xavier Claude, Yannick Naudet, Eric Tobias, Angeliki Antoniou, George Lepouras, and Costas Vasilakis. 2013. Improving museum visitors' Quality of Experience through intelligent recommendations: A visiting style-based approach. In Workshop Proceedings of the 9th International Conference on intelligent environments. IOS Press, 507-518
Elmar Mair, Gregory D Hager, Darius Burschka, Michael Suppa, and Gerhard Hirzinger. 2010. Adaptive and generic corner detection based on the accelerated segment test. In European conference on Computer vision. Springer, 183-196
Karim Malik, Colin Robertson, Steven A Roberts, Tarmo K Remmel, and Jed A Long. 2022. Computer vision models for comparing spatial patterns: understand-ing spatial scale. International Journal of Geographical Information Science (2022), 1-35
R. Mayer and S. Sheehan. 1991. The Artist's Handbook of Materials and Techniques. Viking. https://books.google.lu/books?id=tQ9nYreyGwEC
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform man-ifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 2018
Pablo Messina, Manuel Cartagena, Patricio Cerda, Felipe del Rio, and Denis Parra. 2020. CuratorNet: Visually-aware Recommendation of Art Images. (2020
Pablo Messina, Vicente Dominguez, Denis Parra, Christoph Trattner, and Alvaro Soto. 2017. Exploring Content-based Artwork Recommendation with Metadata and Visual Features. arXiv preprint arXiv:1706.05786 (2017
Pablo Messina, Vicente Dominguez, Denis Parra, Christoph Trattner, and Alvaro Soto. 2019. Content-based artwork recommendation: integrating painting meta-data with neural and manually-engineered visual features. User Modeling and User-Adapted Interaction 29, 2 (2019), 251-290
F. Nah. 2004. A study on tolerable waiting time: how long are web users willing to wait? Behaviour & Information Technology 23 (2004). Issue 3
Yannick Naudet, Angeliki Antoniou, Ioanna Lykourentzou, Eric Tobias, Jenny Rompa, and George Lepouras. 2015. Museum personalization based on gaming and cognitive styles: the BLUE experiment. International Journal of Virtual Communities and Social Networking (IJVCSN) 7, 2 (2015), 1-30
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. Automatic evaluation of topic coherence. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics. Association for Computational Linguistics, 100-108
Jakob Nielsen. 2009. Powers of 10: Time Scales in User Experience. https://www.nngroup.com/articles/powers-of-10-time-scales-in-ux/
Eli Pariser. 2011. The flter bubble: What the Internet is hiding from you. penguin UK
Bilih Priyogi. 2019. Preference elicitation strategy for conversational recom-mender system. In Proceedings of the twelfth ACM international conference on web search and data mining. 824-825
Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for recommender systems. In Proceedings of the ffth ACM conference on Recommender systems. 157-164
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th Inter-national Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748-8763. https://proceedings.mlr.press/v139/radford21a.html
Pradeep Rai and Shubha Singh. 2010. A survey of clustering techniques. Interna-tional Journal of Computer Applications 7, 12 (2010), 1-5
Dhanesh Ramachandram and Graham W. Taylor. 2017. Deep Multimodal Learn-ing: A Survey on Recent Advances and Trends. IEEE Signal Process. Mag. 34, 6 (2017), 96-108
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. CoRR abs/1908.10084 (2019). arXiv:1908.10084 http://arxiv.org/abs/1908.10084
Stefen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factor-izing personalized markov chains for next-basket recommendation. In Proceedings of the 19th international conference on World wide web. 811-820
Deepjyoti Roy and Mala Dutta. 2022. A systematic review and research perspec-tive on recommender systems. Journal of Big Data 9, 1 (2022), 1-36
Ayodeji Olalekan Salau and Shruti Jain. 2019. Feature extraction: a survey of the types, techniques, applications. In 2019 International Conference on SignalProcessing and Communication (ICSC). IEEE, 158-164
Holger Schielzeth, Niels J. Dingemanse, Shinichi Nakagawa, David F. Westneat, Hassen Allegue, Céline Teplitsky, Denis Réale, Ned A. Dochtermann, László Zsolt Garamszegi, and Yimen G. Araya-Ajoy. 2020. Robustness of linear mixed-efects models to violations of distributional assumptions. Methods Ecol. Evol. 11, 9 (2020), 1141-1152
Karen Simonyan and AndrewZisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1-9
Eirini Eleni Tsiropoulou, Athina Thanou, and Symeon Papavassiliou. 2017. Qual-ity of Experience-based museum touring: A human in the loop approach. Social Network Analysis and Mining 7, 1 (2017), 1-13
L.J.P. van der Maaten and G.E. Hinton. 2008. Visualizing Data Using t-SNE. J. Mach. Learn. Res. (2008), 2579-2605
Eliséo Véron and Martine Levasseur. 1989. Ethnographie de l'exposition: l'espace, le corps et le sens. Bibliothèque publique d'information du Centre Pompidou
William Webber, Alistair Mofat, and Justin Zobel. 2010. A Similarity Measure for Indefnite Rankings. 28, 4, Article 20 (2010), 38 pages. https://doi.org/10.1145/1852102.1852106
Bereket Abera Yilma, Najib Aghenda, Marcelo Romero, Yannick Naudet, and Hervé Panetto. 2020. Personalised visual art recommendation by learning latent semantic representations. In 2020 15th International Workshop on Semantic and Social Media Adaptation and Personalization (SMA. IEEE, 1-6
Bereket Abera Yilma, Yannick Naudet, and Hervé Panetto. 2021. Personalisation in Cyber-Physical-Social Systems: A Multi-Stakeholder Aware Recommendation and Guidance. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization (Utrecht, Netherlands) (UMAP '21). Association for Computing Machinery, New York, NY, USA, 251-255. https://doi.org/10.1145/3450613.3456847
Feng Zhao, Yajun Zhu, Hai Jin, and Laurence T Yang. 2016. A personalized hashtag recommendation approach using LDA-based topic model in microblog environment. Future Generation Computer Systems 65 (2016), 196-206
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. 2021. Topic modelling meets deep neural networks: A survey. arXiv preprint arXiv:2103.00498 (2021).