Why do I like this Song?

From 2009 I led a music startup called We Are Hunted that was moderately successful. It was one of the first music charts powered by social media and was a popular tool for music discovery. It was later acquired by Twitter in 2012. I was one of the tech leads on Twitter Music in 2013/2014. Since leaving Twitter I've been thinking a lot about new models for music discovery. What follows is a summary of the research I found most useful. I share in the hope that someone else might find it useful too. Enjoy!

Stephen Phillips @huntedguy
30 April 2015

Intelligent Music Software

The music industry has grown through evolutions of media. From physical formats like LPs and CDs to digital downloads and streaming. For fans, this means cheaper music, bigger catalogs and a range of devices. Today music fans are listening to more music than ever before. This growth demands ever more intelligent software that delivers better music experiences.

Determining song similarity is the base function of intelligent music discovery. Given a song or artist as a starting point, a similarity function is used to generate song playlists, determine related artists, or make personal recommendations.

There are several techniques for determining song similarity:

Expert - A music expert reviews each song against a set of descriptive features. These features are used to group related songs together. This technique produces really high quality playlists, but is difficult and expensive to scale.

Social - Use listening history to group songs together. People who like The Killers, also like The Strokes. Collaborative filtering produces excellent playlists for popular music, but is poorer for new music as it lacks sufficient history.

Acoustic - Group songs by their acoustic properties like tempo, timbre, melody and more. Calculating these features is technically challenging, but works really well as a similarity measure. It can perform poorly in certain situations as it lacks context like artist personality and popularity.

Mixed - Use a combination of these techniques in a way that favors their strengths and minimizes their weaknesses. At We Are Hunted, we wrote software that searched song reviews on music blogs (experts) to find the most common descriptive phrases (social). This approach had the positives of both techniques, in a way that would scale, with the ability to capture new music as soon as it was released.

Music Taste is Subjective

No matter which technique is used, there is a quality ceiling of about 80%. At best, one in five songs recommended are not considered a good choice by the listener. Worse still, different people select different songs as being the odd one out. This points to a major weakness in each approach. They all focus on the song and largely ignore the listener.

While a song has objective attributes that can be measured, it is the subjective attributes, unique to each listener, that ultimately determines likability.

Perhaps we haven't been measuring the right thing. Comparing songs based on expert or acoustic features is like judging a painting by the colors used. Or counting the number of feathers on a jet plane to predict if it will fly. New and better techniques may come from a deeper understanding of why a listener likes a song.

A simple question, 'why do I like this song?', proves to be very tough to answer. It reaches into the heart of the human experience. You soon find yourself learning about the physics of sound, the anatomy of the ear, the chemistry and neurology of the brain, evolution and psychology. I recently spent a few weeks consuming as much research as I could find, and my simple conclusions are offered below as an answer to this question.

Music is a Game

When we listen to a song, and we are able to correctly identify and resolve it's patterns our body rewards us by releasing chemicals that make us feel pleasure. We remember songs that make us feel good.

The more we listen to a song, the more the song becomes imprinted in our memory. The songs we like most are the songs we have heard the most before.

Music is a Drug

This chemical response is similar to other drugs. On first listen we gain some pleasure. On subsequent listens pleasure increases, but only to a point. Eventually our body adapts and the pleasure we receive diminishes. We get tired of listening to the same song. Over time this adaptation dissipates while the memory lives on. If we return to an old song we like, we immediately feel heightened pleasure but also more quickly become tired of it.

When combined with other drugs that give pleasure, the music experience is amplified. Yes that song does sound better when you are high.  

New Music isn't New

We like new music that is similar to the music we have heard the most before, but with some new novel elements. Novel enough that it challenges us to play a new game, but not too different to make us confused, and lose the game.

We dislike new music that is very different to the music we already know, because we have not learnt how to correctly resolve it's patterns. We can learn and appreciate new forms of music through extended listening.

Music is Sexy

Evolution theorists suggest music was selected as a mating advantage. Males who could sing attracted the best females. Teenage girls are forever destined to adore pop stars. Love songs will remain the most popular topic for songwriters.

Music Binds Emotions

When we like a song and store it in our memory, it is stored along with all the emotions we felt at that time. On subsequent listens, these emotions are returned. This is why some songs can give pleasure, while also making us sad.

Music Develops Early

We develop our taste in music in our teens and early adult life. During this period our brain stores memories much more strongly than at any other time in our life. We should be able to accurately predict a person's age from their favorite songs.

Music Reflects

Women talk to women more than men, and therefore prefer the voice of women in songs. Likewise people prefer singers of similar ethnic origin to themselves.

If you are powerless in your life, powerful music will appeal to you. Teens from broken homes or difficult social environments (bullied) are attracted to the power (bass) of Hip Hop, Metal and Hard Rock.

If you are think you are smart, you will like music that appeals to your intelligence with poetic lyrics and complex melodies.

If you grow up in urban areas, you will like music that mimics the sounds and rhythms of the street.

Music is a Tool

Music is used for many more activities than just listening pleasure. It can be used to help focus the mind for work, study or driving. It can distract us from pain or exhaustion during exercise. It can make us dance and move to the beat.

Music is Small

While most modern music apps boast catalogs of +10 million songs, most people remember fewer than 3,000 songs and have less than 50 favorite songs. Music junkies can remember 5,000 or more.

Music is Movement

Entrainment suggests that music make us move whether we like it or not. Your foot taps and your heartbeat syncs to the beat.

Music is Social

Before recorded music was invented, music was always social. It existed only as live performance. Evolution theorists view music as a primitive language used by communities to share their history.

The impact of music is amplified when experienced with other people. The closer you are to the performer, the more intense the experience. The fervor of live performance enhances the pleasure for everyone.

Music App Design

You could argue that the design of music apps has not evolved much since the early versions of iTunes, Pandora and Rhapsody. While the user experience has clearly improved, and the access model has shifted from download to streaming, they are still primarily optimized for managing playlists and searching huge catalogs of songs.

Designing music apps to be a much more personal listening experience seems an obvious next design innovation.

Initialization - The on-board process for music apps is vital to performance. The app should learn as much about the user as possible. Age, sex, country of origin, city/state as a teen, favorite songs, common activities, and more. This process should be fun and engaging. Music fans have shown they are willing to invest significant time to train and tune their music service.

Recommendation - This deeper knowledge of the listener's music experience will allow the app to better know what new songs to recommend and when to play an old song at just the right time. Automated playlists can reach new levels of quality.

Activity - Whether it be focus, distraction or movement, playlists should always adapt to the current activity of the listener.

Control - Voice control is a huge part of the future design of music apps. Nearly every popular use case for music consumption demands your hands doing other things besides typing or swiping. Mind control will be here even sooner than we think and supplement voice as a feedback loop for optimizing playlists.

Mobile - Likewise, popular use cases for music consumption are mostly mobile. Whichever app has the best mobile experience will dominate the market. It is not difficult to imagine a successful music company with no desktop product at all.

Similarity - Existing similarity measures will remain vital tools for music recommendation. These techniques will continue to evolve and improve, to speed the capture of detailed personal profiles.

Popularity - Measuring popularity remains an effective filter for discovery. It acts as a proxy for song quality, which is difficult (if not impossible) to determine using the objective features of a song.

Age - Determining the release date of track remains a very difficult problem at scale. Combining release dates with popularity metrics is vital to making accurate suggestions for user based on their age.

Location - Likewise understanding where a song is popular, is essential for making accurate personal recommendations.

Into the Future

Social

While many music apps have explored social models for discovery, it's biggest impact will see social media transform the music experience itself. Social music can bring us closer to the musicians we love, and make it easier than ever to share the experience with fans like ourselves. A virtual live performance to an audience of millions worldwide remains both compelling and elusive. 

So far we have made very little progress. The best we have seen was Turntable.fm in 2011, who generated an excitement for social music that has not been felt since. They created something magical, that for a brief moment in time, shined a light on the future. It was social, it was addictive, and it was hella fun. 

Holograms

The music industry will again lead popular culture as Holograms go mainstream. Holographic music videos (Holos?) will give music fans an amazing new sense of immersion. Your favorite band will play live in your town (and ultimately your living room) every night of the week.

Neuroscience

And finally, in coming years there will be lots of fun to be had, better understanding what our brains are doing when listening to music. The tools exist already to help all of us get started.

Music and Emotion - Wikipedia

The study of music and emotion seeks to understand the psychological relationship between human affect and music. It is a branch of music psychology with numerous areas of study, including the nature of emotional reactions to music, how characteristics of the listener may determine which emotions are felt, and which components of a musical composition or performance may elicit certain reactions. The field draws upon and has significant implications for such areas as philosophy, musicology, and aesthetics, as well the acts of musical composition and performance.

Leonard Bernstein teaches the Harmonic Series

http://en.wikipedia.org/wiki/Leonard_Bernstein

The Harmonic Series is the foundation for all the world's music, it's vast variety and richness. There isn't any pitch in any music system in any culture that isn’t found in the harmonic series. 

The Harmonic Series is the foundation for all the world's music, it's vast variety and richness. There isn't any pitch in any music system in any culture that isn’t found in the harmonic series. Tangomusicology

Emotion and Meaning in Music - Leonard B. Meyer

http://www.amazon.com/Emotion-Meaning-Music-Phoenix-Books

Emotion and Meaning in Music by Leonard B. Meyer is a landmark in the history of music theory. It links theoretical arguments concerning music explicitly to human psychology. It begins with the premise that musical experience is a species of human psychology, and as such, music theory benefits from an understanding of psychology.

"Altogether it is a book that should be required reading for any student of music, be he composer, performer, or theorist. It clears the air of many confused notions . . . and lays the groundwork for exhaustive study of the basic problem of music theory and aesthetics, the relationship between pattern and meaning."

Meyer regards learning as the overwhelmingly most important phenomenon underlying musical expectations and experience.

Musical styles are more or less complex systems of sound relationships understood and used in common by a group of individuals. Styles are learned sets of expectations. Styles provide the norms against which ensuing musical events can be heard by a listener as expected or unexpected. Meyer regards the probability of events as key contributors to the sense of style and looks favorably on statistical style studies.

Leonard B. Meyer - Wikipedia
http://en.wikipedia.org/wiki/Leonard_B._Meyer

Ohio State University - School of Music
http://www.music-cog.ohio-state.edu/Music829D/Notes/Meyer1.html

David Huron and the Science of Sound

Bertrand Russell - Wikipedia

http://en.wikipedia.org/wiki/Bertrand_Russell

Bertrand Russell believed that understanding music is not a matter of dictionary definitions, of knowing this, that, or the other rule of musical syntax and grammar, rather it is a matter of habits correctly acquired in one's self and properly presumed in the particular work... the habits acquired are not universal but are acquired in connection with a particular style and are relevant to that particular style.

Certain musical relationships appear to be well-nigh universal. In almost all cultures, for example, the octave and the fifth or fourth are treated as stable, focal tones toward which other terms of the system tend to move. Similarly many systems have organized tonal progressions, scales, though the relationships between these sound stimuli will vary greatly from system to system.

Robert Zatorre / Anne Blood

In 2001, neuroscientists Anne Blood and Robert Zatorre at McGill University in Montreal used magnetic resonance imaging to show that people listening to pleasurable music had activated brain regions called the limbic and paralimbic areas, which are connected to euphoric reward responses, like those we experience from sex, good food and addictive drugs. Those rewards come from a gush of a neurotransmitter called dopamine. As DJ Lee Haslam told us, music is the drug.

Music sets up sonic patterns and regularities that tempt us to make unconscious predictions about what's coming next. If we’re right, the brain gives itself a little reward – as we’d now see it, a surge of dopamine. The constant dance between expectation and outcome thus enlivens the brain with a pleasurable play of emotions.

In order to have any expectations about where the music will go in the first place, you need to know the rules – to appreciate what is normal. This varies from one culture to another.

Music can provoke other feelings too, such as anxiety, boredom and even anger. Composers need to tweak expectations to just the right degree. Not enough, and the music is dully predictable, as nursery tunes seem to adults. Too much, and we can't develop any expectations at all – which is why many people struggle with modernist atonal music.

Will we ever understand why music makes us feel good? - BBC

If music-induced emotional states can lead to dopamine release, it may begin to explain why musical experiences are so valued. These results further speak to why music can be effectively used in rituals, marketing or film to manipulate hedonistic states. Our findings provide neurochemical evidence that intense emotional responses to music involve ancient reward circuitry and serve as a starting point for more detailed investigations of the biological substrates that underlie abstract forms of pleasure.

http://www.theguardian.com/science/2011/jan/09/why-we-love-music-research

Why Does Music Give Us Chills?

About 50 percent of people get chills when listening to music. Research shows that's because music stimulates an ancient reward pathway in the brain, encouraging dopamine to flood the striatum—a part of the forebrain activated by addiction, reward, and motivation. Music, it seems, may affect our brains the same way that sex, gambling, and potato chips do.

Strangely, those dopamine levels can peak several seconds before the song’s special moment. That’s because your brain is a good listener—it’s constantly predicting what’s going to happen next. Evolutionarily speaking, it’s a handy habit to have. Making good predictions is essential for survival.

But music is tricky. It can be unpredictable, teasing our brains and keeping those dopamine triggers guessing. And that’s where the chills may come in. Because when you finally hear that long awaited chord, the striatum sighs with dopamine-soaked satisfaction and—BAM—you get the chills. The greater the build-up, the greater the chill.

The psychological functions of music listening

People listen to music to regulate arousal and mood, to achieve self-awareness, and as an expression of social relatedness. The first and second dimensions were judged to be much more important than the third - a result that contrasts with the idea that music has evolved primarily as a means for social cohesion and communication.

The psychological functions of music listening

Yes, certain songs are better at provoking that feeling of omnipotence than others, and the research team says it's a question of bass level.

Why Does Music Make Us Feel?

We are led to the conclusion that the artifact of music should contain some distinctly human elements. The question, of course, is what those elements are. One candidate is our expressive speech – perhaps music is just an abstract form of language.

Regardless of whether music is emotional intonation from speech or a summary of expressive movements – or something else altogether – new research adds yet more fuel to the expectation that music has been culturally selected to sound like an emotionally expressive human. While it is not easy for us to see the human ingredients in the modulations of pitch, intensity, tempo and rhythm that make music, perhaps it is obvious to our auditory homunculus.

Entrainment in the biomusicological sense refers to the synchronization of organisms to an external rhythm, usually produced by other organisms with whom they interact socially. Examples include firefly flashing, mosquito wing clapping, as well as human music and dance such as foot tapping.

Beat induction can be seen as a fundamental cognitive skill that allows for music. We can hear a pulse in a rhythmic pattern while it might not even be explicitly in there.

Entrainment in music has been suggested as an emotion mechanism where the music causes an increase in arousal "because the powerful, external rhythm of the music interacts with an internal body rhythm of the listener, such that the latter rhythm adjusts towards and eventually 'locks in' to a common periodicity."

The adjusted body rhythm then spreads through proprioceptive feedback to the subcomponents of emotion such as cognitive, neurophysiological, motivational, expressive, and subjective feelings components.

http://cms.unige.ch/fapse/neuroemo/pdf/Labbe%20&%20Grandjean_ARF2011.pdf

http://en.wikipedia.org/wiki/Entrainment_(biomusicology)

Mafa (African Tribe) and Western listeners derive similar emotional meaning from the tempo and key of musical passages. Both groups tended to classify fast-paced pieces as happy and slow ones as scared or fearful, and mostly agreed on which passages were sad, but assigned no particular tempo with them. Mafa and Westerners also generally regarded major-key pieces as happy, minor-key excerpts as fearful and passages with an indeterminate key as sad.

http://www.sott.net/article/179550-Feelings-Universal-Musical-Feelings

The Surprising Science Behind What Music Does To Our Brains

In a study of couples who spent time getting to know each other, looking at each other's top 10 favorite songs actually provided fairly reliable predictions as to the listener’s personality traits.

The study used five personality traits for the test: openness to experience, extraversion, agreeableness, conscientiousness, and emotional stability.

The Neuroscience Of Music

Why does music make us feel? On the one hand, music is a purely abstract art form, devoid of language or explicit ideas. The stories it tells are all subtlety and subtext. And yet, even though music says little, it still manages to touch us deep, to tickle some universal nerves. When listening to our favorite songs, our body betrays all the symptoms of emotional arousal. The pupils in our eyes dilate, our pulse and blood pressure rise, the electrical conductance of our skin is lowered, and the cerebellum, a brain region associated with bodily movement, becomes strangely active. Blood is even re-directed to the muscles in our legs. (Some speculate that this is why we begin tapping our feet.)

We can now begin to understand where these feelings come from, why a mass of vibrating air hurtling through space can trigger such intense states of excitement. A brand new paper in Nature Neuroscience by a team of Montreal researchers marks an important step in revealing the precise underpinnings of "the potent pleasurable stimulus" that is music.

The scientists found that our favorite moments in the music were preceeded by a prolonged increase of activity in the caudate. They call this the "anticipatory phase" and argue that the purpose of this activity is to help us predict the arrival of our favorite part.

The anticipatory phase, set off by temporal cues signaling that a potentially pleasurable auditory sequence is coming, can trigger expectations of euphoric emotional states and create a sense of wanting and reward prediction. This reward is entirely abstract and may involve such factors as suspended expectations and a sense of resolution. Indeed, composers and performers frequently take advantage of such phenomena, and manipulate emotional arousal by violating expectations in certain ways or by delaying the predicted outcome (for example, by inserting unexpected notes or slowing tempo) before the resolution to heighten the motivation for completion. The peak emotional response evoked by hearing the desired sequence would represent the consummatory or liking phase, representing fulfilled expectations and accurate reward prediction. We propose that each of these phases may involve dopamine release, but in different subcircuits of the striatum, which have different connectivity and functional roles.

While music can often seem (at least to the outsider) like a labyrinth of intricate patterns – it's art at its most mathematical – it turns out that the most important part of every song or symphony is when the patterns break down, when the sound becomes unpredictable. If the music is too obvious, it is annoyingly boring, like an alarm clock.

Numerous studies have demonstrated that dopamine neurons quickly adapt to predictable rewards. If we know what's going to happen next, then we don’t get excited. This is why composers introduce the tonic note in the beginning of the song and then studiously avoid it until the end. The longer we are denied the pattern we expect, the greater the emotional release when the pattern returns, safe and sound. That is when we get the chills.

Music information retrieval

Analysis can often require some summarising and for music this is achieved by feature extraction, especially when the audio content itself is analysed and machine learning is to be applied. The purpose is to reduce the sheer quantity of data down to a manageable set of values so that learning can be performed within a reasonable time-frame. One common feature extracted is the Mel-Frequency Cepstral Coefficient (MFCC) which is a measure of the timbre of a piece of music. Other features may be employed to represent the key, chords, harmonies, melody, main pitch, beats per minute or rhythm in the piece.

Reminiscence Bump

People form their most potent memories between the ages of 12-22. The reminiscence bump is the tendency for older adults to have increased recollection for events that occurred during their adolescence and early adulthood. It was identified through the study of autobiographical memory and the subsequent plotting of the age of encoding of memories to form the lifespan retrieval curve.

Technology is useful and convenient, but it has, in the end, reduced its own value and increased the value of the things it has never been able to capture or reproduce.

The less likely an event is, the more information it provides when it occurs.

http://www.smithsonianmag.com/arts-culture/how-do-our-brains-process-music-32150302/?no-ist

Timbre is the quality of a musical note or sound that distinguishes different types of sound production or musical instruments. The physical characteristics of sound that mediate the perception of timbre include spectrum and envelope. Timbre is also known in psychoacoustics as sound quality or sound color.http://www.joehallock.com/edu/COM498/preferences.html

System of measuring similarity between two songs

After getting the HPCP feature, the pitch of the signal in a time section is known. The HPCP feature has been used to compute similarity between two songs in many research. A system of measuring similarity between two songs is shown in Fig.3. First, time-frequency analysis is needed to extract the HPCP feature. And then set two songs' HPCP feature to a global HPCP, so there is a standard of comparing. The next step is to use the two features to construct a binary similarity matrix. Smith–Waterman algorithm is used to construct a local alignment matrix H in the Dynamic Programming Local Alignment. Finally, after doing post processing, the distance between two songs can be computed.

http://en.wikipedia.org/wiki/Harmonic_pitch_class_profiles

Last year I bought an EEG headset (the Mindwave Mobile) to play with my Raspberry Pi. I thought it might be possible to have it recognize when you dislike music and then switch the song on Pandora for you. This would be great for when you are working on something or moving around away from your computer.

http://stevenhickson.blogspot.com.au/2014/02/controlling-music-with-your-mind.html

Credits

Title Screen Background 
http://adaptiveart.eecs.umich.edu/2010/blog/2010/11/02/emotion-music/