Using GANs to Generate Lyric Videos

Artificial intelligence; art; expressive artificial intelligence; generative adversarial networks; image generation; video generation Artificial intelligence; video generation; Industrial and Manufacturing Engineering; Environmental Engineering

Abstract :

[en] Artificial intelligence (AI) technologies have become increasingly common in creative practices in recent years. The rising number of research initiatives that emerge at the intersection of AI and art prompts researchers and artists to analyze the creative and explorative applications of AI in the context of art. First, this paper describes a specific AI art piece, an AI-generated music video for a song called Initiation, illustrating how AI can be used for creative purposes. This art piece was created in the context of the opening of Esch2022, the European Capital of Culture. Then, the paper provides an overview of the potential implications of AI technologies on the general understanding and creation of art.

Disciplines :

Computer science

Author, co-author :

GAREEV, Daniel ; University of Luxembourg > Faculty of Science, Technology and Medicine > Department of Computer Science > Team Leon VAN DER TORRE

GLASSL, Oliver ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > FSTM Faculty administration

NOUZRI, Sana ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Computer Science (DCS)

External co-authors :

Language :

English

Title :

Using GANs to Generate Lyric Videos

Publication date :

2022

Journal title :

IFAC-PapersOnLine

ISSN :

2405-8971

eISSN :

2405-8963

Publisher :

Elsevier BV

Volume :

Issue :

Pages :

3292-3297

Peer reviewed :

Peer Reviewed verified by ORBi

Additional URL :

https://api.elsevier.com/content/article/PII:S240589632202136X?httpAccept=text/xml

Funders :

Université du Luxembourg

Available on ORBilu :

since 23 May 2024

Statistics

Number of views

52 (5 by Unilu)

Number of downloads

50 (0 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

Brock, A., Donahue, J., and Simonyan, K. (2018). Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096.
Cetinic, E. and She, J. (2022). Understanding and creating art with ai: Review and outlook. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 18(2), 1-22.
Chollet, F. (2017). Xception: Deep learning with depth-wise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1251-1258.
Donahue, J. and Simonyan, K. (2019). Large scale adversarial representation learning. Advances in Neural Information Processing Systems, 32.
Frosst, N., Kereliuk, J., and Kid, G. (2019). Text conditional lyric video generation. In chap. Machine Learning for Creativity and Design Workshop.
Gareev, D. (2022). Stylegan2 repository. URL https://github.com/lowlypalace/StyleGAN2.
Gareev, D. and Glassl, O. (2022). Initiation lyric video. URL https://youtu.be/_JVv8CtXVKA.
Glassl, O. and Savo, M. (2019). Initiation. URL https://distrokid.com/hyperfollow/thalamusproject/initiation.
Glassl, O. and Savo, M. (2022). Thalamusproject. URL https://www.thalamusproject.com/.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Advances in neural information processing systems, 27.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770-778.
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. (2020a). Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33, 12104-12114.
Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T. (2021). Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 34.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020b). Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8110-8119.
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 8748-8763. PMLR.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. In International Conference on Machine Learning, 8821-8831. PMLR.
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2818-2826.
Torrico, T.J. and Munakomi, S. (2019). Neuroanatomy, thalamus.
Varrette, S., Bouvry, P., Cartiaux, H., and Georgatos, F. (2014). Management of an academic hpc cluster: The ul experience. In Proc. of the 2014 Intl. Conf. on High Performance Computing & Simulation (HPCS 2014), 959-967. IEEE, Bologna, Italy.
Ward, L.M. (2013). The thalamus: gateway to the mind. Wiley Interdisciplinary Reviews: Cognitive Science, 4(6), 609-622.
Wikipedia (2022a). Esch 2022. URL https://lb.wikipedia.org/wiki/Esch_2022.
Wikipedia (2022b). Unsplash. URL https://en.wikipedia.org/wiki/Unsplash.
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018). Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1316-1324.