Over the past few decades, computer scientists have developed computational tools that can generate specific types of data, such as images, words or audio recordings. These systems could have a variety of valuable applications, particularly in creative fields that entail the production of new and unique artworks.
Researchers at University of Waterloo have recently developed LyricJam, an advanced computational system that can generate lyrics for live instrumental music. This system, presented in a paper set to be presented at the International Conference on Computational Creativity and pre-published on arXiv, could help artists to compose new lyrics that match well with the music they create.
“I have always had a deep love of music and an interest in learning about the creative processes behind some of my favorite songs,” Olga Vechtomova, one of the researchers who carried out the study, told TechXplore. “This drew me to do research on music and lyrics and how machine learning could be used to design tools that would inspire music artists.”
Vechtomova and her colleagues have been conducting research focusing on lyric generation for a few years now. Initially, they developed a system that can learn specific features or aspects of an artist’s lyrical style, by analyzing audio recordings of their songs and lyrics they composed in the past. This system then uses the information gathered in its analyses to generate lyrics that are aligned with the style of a particular artist.
More recently, the researchers also started investigating the possibility of generating lyrics for audio clips of recorded instrumental music. In their new study, they tried to take this a step forward, by developing a system that can generate suitable lyrics for live music.
“The goal of this research was to design a system that can generate lyrics reflecting the mood and emotions expressed through various aspects of music, such as chords, instruments, tempo, etc.,” Vechtomova said. “We set out to create a tool that musicians could use to get inspiration for their own song writing.”
Essentially, Vechtomova and her colleagues set out to create a system that could process raw live music played by an individual musician or a band and generate lyrics that match the emotions expressed by the music. The artists would then be able to review these generated lyrics and draw inspiration from them or adapt them, thus discovering new interesting themes or lyric ideas that they had not considered before.
“The scenario we envisioned is of an AI system that acts as a co-creative partner with a musician,” Vechtomova explained. “From a user’s perspective the LyricJam app is very simple: a music artist plays live music and the system displays lyric lines that it generates in real time in response to the music it hears. The generated lines are saved for the duration of the session, so that the artist can look at them after they finish playing.”
The system created by the researchers works by converting raw audio files into spectrograms and then using deep learning models to generate lyrics that match the music they processed in real time. The model’s architecture is comprised of two variational autoencoders, one designed to learn representations of music audio and the other to learn lyrics.
Vechtomova and her colleagues then designed two new mechanisms that align the representations of music and lyrics processed by the two autoencoders. Ultimately, these mechanisms allow their system to learn which types of lyrics go well with a particular instrumental music.
“We let the machine learn these associations from the data in an unsupervised way,” Vechtomova said. “As a result, the machine learns the lyrical themes, words and expressions that are associated with different types of music. For instance, we observed that lyrics generated for calm and ambient music are very different than those generated for more aggressive sounding music.”
The main characteristic that sets LyricJam apart from other lyric generation systems developed in the past is that it can create suitable lyrics in real time, as an artist is playing live music. Musicians and other users interested in trying the system can access a live version at https://lyricjam.ai.
“I want to highlight that the core motivation for this research is not to write a song for the artist, but to inspire the artist’s own creativity by suggesting fresh new ideas and expressions that the system generated by hearing their music,” Vechtomova said. “We don’t want to make the lyric writing process easier or faster. Instead, we want to make it more fulfiling, helping artists to get into the creative flow and realize their own creativity by collaborating with the system.”
To evaluate the system they developed, Vechtomova and her colleagues carried out a user study in which they asked musicians to play live music and share their feedback about the lyrics created by their system. Interestingly, most musicians who took part in this study said that they perceived LyricJam as a non-critical jam partner that encouraged them to improvise and experiment with unusual musical expressions.
“For example, by shifting their musical style or trying new chord progressions, artists participating in our user study observed a real-time change in lyrical themes, which they found encouraging,” Vechtomova said. “This suggests that the system could be useful not just for lyric writing but for improvisation and composing music too.”
In the future, LyricJam could prove to be a highly valuable tool for musicians and artists worldwide, helping them to compose unique and interesting lyrics for their songs. Vechtomova and her colleagues are currently working on a final version of the system that could be easily accessed by artists worldwide, while also trying to design other tools that could improve lyric writing processes.
“Designing tools that help music artists to unlock their creativity is one of my core research interests,” Vechtomova said. “I have a number of ongoing research projects in my lab, where we look at other aspects of lyrics and music, such as phonetic characteristics of lyrics and the musical structure of songs, so that we can potentially incorporate these aspects into lyric generative models.”
A system to generate new song lyrics that match the style of specific artists
More information:
LyricJam: A system for generating lyrics for live instrumental music. arXiv:2106.01960 [cs.SD]. arxiv.org/abs/2106.01960
2021 Science X Network
Citation:
LyricJam: A system that can generate lyrics for live instrumental music (2021, June 28)
retrieved 28 June 2021
from https://techxplore.com/news/2021-06-lyricjam-lyrics-instrumental-music.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.