AI Talk: Recommender systems

Oct. 30, 2020 / By V. “Juggy” Jagannathan, PhD

This week’s AI Talk…

My friend sent me an article a few weeks ago on how Spotify recommends music to its listeners. It was an excellent article and it got me thinking about recommender systems. We are completely surrounded by these systems. If you watch something on Amazon, Netflix or YouTube, you immediately get a list of recommendations on what to watch next. YouTube will only give you a few seconds to decide before it automatically plays its recommendation. And, it is not just music or videos. I constantly get recommendations on Amazon—people who bought this book also bought this other book. Same goes for restaurant recommendations. Then there are targeted advertisements, a manifestation of similar technology that assesses your personal profile and figures out what to show you.

So, how do these systems work? Spotify, it turns out, is holding its own amidst technology heavy hitters from Apple to Amazon to Google. Spotify’s machine learning models use a variety of approaches to curate personalized playlists for each and every one of its millions of subscribers.

Collaborative Filtering

One of the primary approaches recommender systems use is a technique called “collaborative filtering.” This is a fairly straightforward comparison of user preferences within a specific database of users. If your likes and preferences are similar to another user, then whatever the other user likes which you have not yet explored becomes a recommendation for you. Netflix, Amazon and others routinely use this fairly simple trick.

Content-based Filtering

There is more recommender systems can do by analyzing the content that you enjoy. For instance, Spotify will use a Convolution Neural Network (a deep learning technique) to extract features from the music you listen to, like the beats in the song. It then determines other songs similar to the song you like. This technique is referred to as content-based filtering.

Spotify also uses published music reviews and assiduously collects and assembles all information related to its song collection. It uses this text to determine the likelihood a person who likes one genre or music track will like another. This is another aspect of content-based filtering.

Open Challenges

Finally, there is the idea of crowdsourcing for a solution. Spotify promotes its Spotify Million Playlist Dataset Challengewhere anyone can compete to develop an algorithm which will predict the next song in a playlist. This is a great way to get opensourced recommender systems. What if they don’t disclose any of the systems they actually end up using? These types of challenges are actually quite common in the machine learning community. Netflix had one such challenge awhile back and it awarded a $1 million grand prize a decade ago to a team which significantly improved its recommendation system. Academia continuously organizes such challenges. To look at the latest challenge and datasets available for building recommender systems, check out the RecSys Challenge 2020.

I used to get annoyed at recommendations provided by these systems, but these days I am noticing that more and more appear to be interesting choices. Not really sure if that is a good or bad thing!


My long-time friend Chandy sent the Spotify article link.

I am always looking for feedback and if you would like me to cover a story, please let me know. “See something, say something!” Leave me a comment below or ask a question on my blogger profile page.

V. “Juggy” Jagannathan, PhD, is Director of Research for 3M M*Modal and is an AI Evangelist with four decades of experience in AI and Computer Science research.