Using Generative Algorithmic Music to Explore Emotive Musical Expression

Music and emotions do not need to be explained to a person, as generally we have a fundamental understanding given by our senses and mental processing systems. However for a computer to generate expressive music based on rules, this ‘understanding’ needs to explained meticulously. We may even find that our ‘understanding’ is partly informed by unconscious and subjective processing, making it more difficult to pinpoint. The process of the study of musical expression will be described throughout this blog.

Forming Musical Ideas

I have studied music and computing in many different ways for many years, so the combination of the two feels like a natural step to take for me. Previously I have attempted to develop a computer program to harmonise chorale melodies in the style of Bach. This is because of a long time fascination of the relationship between the musical parts and the expression. A wide variety of music is capable of making me feel emotions. This made me very curious about the composition process and manipulation of sound. Having learnt multiple instruments and composing bits and pieces, I had a useful foothold on the subject area.

Image from https://michaelkravchuk.com/free-piano-sheet-music-bourree-in-e-minor-bach/

Having read about famous 20th century music teachers Nadia Boulanger and Arnold Schoenberg, and music composition pedagogy or lack thereof. It became clear I would not find answers I was looking for there. This is not to say Boulanger or Schoenberg were bad teachers, or that music composition pedagogy does not exist, but I was looking for study of musical features and their relationship to expression rather than music theory. This led me to research on music origins, philosophy and psychology, in an attempt to understand how something as seemingly abstract as vibrations and ratios can express emotion.

Summary of Academic Music Research

Each discipline adds perspective to the study of musical expression. How sound affects our emotions.

Music Origins

This highlighted sound and musics function, particularly in language. Music shares a staggering amount of similarities with language, leading multiple researchers to propose the theory that music and language has a single shared precursor. Of course humans are not the only animals with a musical language. Music also serves a social function, it is used in cafes, shops, church, birthday parties and on transport. Steven Mithen provides a model framework of music and speech processing systems in the brain, of which many were shared. An important clue to attempt to answer for musical expression, is that for both music and speech processing, we analyse musical features and conclude about the sounds expression, maybe unconsciously.

Music Philosophy

This study attempts to account for musical expressiveness with different theories. One theory suggests music is symbolic of things such as old texts, events, typical emotions behavioural displays and nature. Another theory suggests the music carries the expression of the composer or performer. In contrast to this it is also suggested that the music may evoke or reflects occurring emotions in the listener. It may also be that music is expressive without the need of occurring emotions and it has expressive character like visual art. Some even believe that music is a language of emotions universally recognised. This is not the case seen in cross cultural study, however some aspects of language, such as how a parent communicates with a child, has been observed to be similar and somewhat universal. Music philosopher Steven Davies points out the importance of large scale features, such as melodic development.

Music Psychology

This area attempts to account for musical features in terms of expression. For example there are studies on the importance of timbre, tonality and melodic contour in perceived expression. Psychology studies also look at the similarities and differences of music and language reinforcing their link. Music therapy is also an important area of study, showing how music is linked to our motor systems and emotions. One major source written by Alf Gabrielsson describes how each indentified musical feature may suggest a certain expression. Music expression is complicated and relies on many factors. Each discipline has many valid things to say. For this reason my next move was to interview two professional classical musicians, as I felt performers would be able to provide unique perspective, as people who work with music for a living.

Conversation with Musicians
and Forming Ideas

Following the literature review I decided to discuss some of these ideas with performers and propose questions.

To address points of controversy and broaden the perspective of the literature review, I contacted two professional British provincial orchestra violinists and communicated over multiple texts and calls. The performers have decades of experience between them, playing in London and abroad.

Firstly we discussed the similarities in music and language. This topic is important because it strengthens the notion music is functional and a fundamental, important part of life for most people. This is in contrast to unattractive reductions of music that are sometimes present. I do not believe it is merely by chance or unimportant that music has vast emotive capacity. Mithen’s highlighting of the existing theory that music and language have a single predecessor, gives a more sufficient framework to explain how music is emotive and why every person has a baseline musicality.
Both musicians were very aware of the similarities in music and speech. One performer used comedic timing as an example of rhythm and stress in speech. The other performers first language is Cantonese, having spent childhood in Singapore. They explained Cantonese is a tone language, and they must have been aware of that musical feature of language from a young age but were unable to say whether it gave them any advantage. In my experience, growing up in greater London, it seems to me the musical features of language are overlooked. Similarly, people do not think of music in terms of its shared features with language. Including myself, for a while. After studying different music disciplines for 13 years, the extent of music and language similarities has only now become something that has seriously drawn my attention. Thinking retrospectively, there was an acute awareness as I remember saying a piece of music had a particularly effective sound because it resembled wailing. I have also heard country guitar being described as producing a ‘crying’ sound. Another example is how musicians are aware of how ‘vocal like’ their instruments are. The tuned piano has fixed notes in a number of keys, that once the sound is produced it can not be controlled, making it not very vocal like, even with the aid of grace notes and other technique. Some instruments groups such as woodwind or strings, allow the performer to fine tune as they are playing, and control the note, in terms of volume for example, after it has started, making it more vocal like. An instrument is vocal like to the extent it can control prosodic features, words aside.
The acquisition of these skills happens through childhood development. To generalise, in terms of music and language, children listen, learn and practice. It may be circumstantially dependent, considering different levels of exposure to music or language, however most people have a sufficient knowledge of language and music. To illustrate the importance of prosody (a quick definition of which is the amplitude, frequency and duration of consonant and vowel sounds) lets look at sarcasm as an example. Melanie Glenwright, University of Manitoba, studied children’s ability to understand sarcasm and concluded that “Kids detect sarcasm at about age 6, but don’t begin to see the intended humour until around age 10,”. This tells us it is a learned behaviour, which I believe uses music analysis systems in the brain. The reason sarcasm is an excellent example to explore the musical part of speech, is that it requires a knowledge of segmental and suprasegmental features of language. These two linguistics terms describe consonant and vowel sounds (segmental), and the features of speech such as stress and intonation (suprasegmental/prosodic), which are really constructed from pitch, rhythm and volume. Stress could be viewed as use of amplitude and duration (dynamic/volume and rhythm) and intonation uses frequency and duration (pitch and rhythm). Sarcasm is traditionally defined ‘as a way of using words’ which are ‘clearly’ intended to convey the opposite meaning, or criticise, or be humorous. For most peoples purposes this needs no further definition, despite the fact it does not tell you about the use of language in much detail. Could an alien learn to be sarcastic with that definition? Having said this, it is difficult to also define sarcasm in terms of prosody, because it is context dependent, however it is clear that it takes more than words alone to express meaning. Consonants and vowels combine to create words which are referential. The sound made when saying dog, does not really have anything to do with dogs, but the sound is the verbalisation of the word, which we understand to have meaning. When we are sarcastic, we make sounds which refer to something we do not actually mean. It is the way we say it that makes people aware of sarcasm; the prosodic features communicate beyond the referential meaning of the words. Every day conversation although mostly mundane, is packed full of melody. Humans use pitch, rhythm and volume to create a wide range of emotive expression in speech. We craft melodies alongside referential sounds to express or hide emotions, often unconsciously. Hopefully that puts the ‘auditory cheesecake’ argument to bed for good. Most people would not class themselves as musical, but it is likely they enjoy music and are not monotonous. I believe in part this is why music has the ability of evoking strong emotions in most people. We have an inherent understanding of melody and it’s so normal to us, it’s easy to overlook. Interestingly Yehudi menuhin, a world renowned violinist, author and more, once wrote, when describing the practice of yoga, “Reduced to our own body, our first instrument, we learn to play it, drawing from it maximum resonance and harmony”. Maybe more research in human evolution would illuminate other features of music. The evolutionary perspective on music is far from fully discussed. Research into the origins of music, highlights much more than language. Bipedalism is suggested to have had multiple affects, the first is the changing of the human body as a result of this development, specifically the throat and effect on vocal chords. In addition Mithen suggests that bipedalism required the evolution of mental mechanisms to maintain the rhythmic coordination of muscle groups. This is an example of complex rhythmic patterns in the body, walking on two feet is possible by acquiring a level of mastery over rhythm and timing. Interestingly the rhythm produced by walking (left, right, left, right etc.) is the first drum rudiment, single stroke. The musical rhythms required to create this left right pattern in walking and drumming, are much more complex than the rhythms produced. This makes me wonder whether research on the heart or lungs from a musical perspective could teach us about our relative understanding of rhythm. The beating of the heart, or expansion and contraction of the diaphragm are seemingly simpler rhythms than that required for movement, or tool use. It is interesting the classic Italian terms for tempo, include andante and corrente (walking and running). Words to describe rhythm such as pace and pulse, are biological functions.

The next set of questions I proposed were whether technical ability had everything to do with interpreting music? Could a player be technically brilliant but lack the ability to give a well received interpretation? For example the renowned pianist Vladimir Horowitz arranged Pictures at an Exhibition and while his performances where well received, his arrangement was not. Although arrangement may have different aspects to the most common type of interpretation of a piece. Before discussing this topic, some context will be given.

Research conducted by Patrik N. Juslin shows how essential prosody is to perceived emotion. He was referenced by Stephen Davies, music philosopher, as believing the performer ‘breathes expression’ in to the music. Similarly expression theory is also mentioned by Davies, which states the music’s expressiveness is dependent on the emotions of the composer or performer at that time and place. This made me think of the transmission of musical ideas. I think it is interesting to view music as a concept or idea that undergoes codifying and translation. Take Bach for example, the original idea was formulated hundreds of years ago codified in original transcriptions, possibly revised and given to musicians to perform. Bach is taking the role of composer codifying his musical ideas to notation. This may then be interpreted by performers who realise the code by translating it to motor movements. An extremely specified and accurate sequence of motor movements, seemingly spontaneously performed, although in reality it may be excruciatingly prepared. And finally, it is heard by listeners. Looking at this ‘Composer > Performer > Listener’ model simply in terms of classical music, most live music performances are in the absence of the composer, by capable performers, to a generally uncapable audience who will have a varying degree of musical education and ability. It is important to remember music is a temporal phenomenological experience, it exists over a period of time where sound is produced and interpreted by the brain. The sheet music is not the music but a coded representation, maybe you could even take that level of abstraction further and say the music is a representation of the ideas or feelings of the composer, such as expression theory suggests. This would also equate music to a form of communication, capable of narrative using sound, similar to universal theory discussed by Davies, where music is the language of emotions. This model also helps explore the features of music. What does the composer define, and what does the performer add? Is viewing the composer as the realising segmental and performer as realising the suprasegmental accurate? Samuil Feinberg, accomplished Russian Soviet composer and pianist, wrote about the duality of the composer and performer in music. Interestingly he writes (http://math.stanford.edu/~ryzhik/Feinberg1.html#:~:text=The%20composer%20needs%20an%20intermediary,creative%20process%20of%20musical%20interpretation.) < expand

The performers felt technical ability is key to interpret music. It seems true that, the higher the technical mastery of an instrument; the wider the range of techniques available for interpretation. However the performers believed there to be a level of subjectivity in interpretations, which may be taught, but are ultimately the result of someone’s life experiences, considering music, emotions and memories. What about people who are famously virtuosic and invent studies or techniques? Is that the result of attaining a certain rare level of mastery, or having a better ability to imagine and interpret sound than other contemporary composers, or both? Of course musicians such as Paganini and Liszt have had a larger impact, that we know of, on the study of music and their given instrument than most composers ever. What divides them from the rest of us? I am not going to seriously try to answer this question, as I would be unable to consider all the relevant factors. However for this study, the reasons for the division of these composers from most, may be in the mastery of the role of composer as well as performer, creating a ‘written and directed’ synergy. It may be mastery of technical ability as well as being better able to imagine and create sound. One performer commented on the difficulty of putting years of technique and experience into words. The fact that musical ability relies on unconscious and internalised processes is a key issue in this study that at least needs to be acknowledged as a limitation, and possibly provide further work to attempt to address. This point is also very relevant to the next question

The next question was about what pieces of music the performers would relate to a set of emotions, and how they would play their instrument to express that emotion. It was clear this was a very difficult question, which may be in part because of the way I proposed it. The set of emotions I decided on were Joy, Fear, Anger, Wonder, Nostalgia. They were chosen to explore different emotion categorisations and models. Joy was selected for a control as a commonly used discrete emotion example. Fear and Anger were selected because they are two emotions used an as example to criticise the Russell circumplex model, as fear and anger may occupy a similar part of the 2 dimensional model, while being very distinct emotions. Finally, Wonder and Nostalgia are emotions featured on the ‘Geneva Emotional Music Scales’ (GEMS), created to describe musically evoked emotion. The largest GEMS scale is GEM-45 which highlights 45 consistently chosen terms to describe music, which are then put into 9 categories. Interestingly they condense these into 3 ‘superfactors’, vitality, sublimity and unease.

Algorithm Design

This section will look at the logic implemented on musical features to guide expression. I will also explore the benefits of using technology and algorithms as a creative music solution.

discussion of James Russell Circumplex Model for reference on emotion expression.