Recommender Systems
Addressing the long tail problem and enhancing the recommendation experience for users on the Headspace App
The Choice Between Two Recommendation Systems
Following weeks of diligent work, you finally have a well-deserved evening free to indulge in a nice restaurant experience. Two friends have offered restaurant recommendations. One friend tends to stick to the tried and true, rarely venturing into uncharted territory. Once a joke lands, they will repeat it incessantly with slight variations. While their taste is dependable, their recommendations have never left you thoroughly impressed. This first friend mirrors a recommender system afflicted by the long tail problem; they recommend popular restaurants that are dependable and palatable to the majority but not necessarily tailored.
Conversely, there’s the friend who constantly embraces risk, blurting out ideas in a haphazard manner. You’re always left wondering about their taste preferences. While they’ve suggested some remarkable dining spots, they’ve also thrown in a fair share of dreadful ones, all delivered with the same level of enthusiasm. The second friend is like a recommender system that throws out random suggestions, occasionally leading to pleasant surprises but also disappointments. Whose advice would you follow for the evening?
In my article, “Beyond Accuracy: Embracing Serendipity and Novelty in Recommendations for Long Term User Retention,” I discussed the importance of going beyond mere accuracy metrics to address the long tail problem and enhance the recommendation experience for users. In this article, I will discuss a real-world implementation when I worked as a machine learning engineer at Headspace Health. While there is a paper published under the Creative Commons Attribution 4.0 International License that goes in-depth with the implementation, I’ll explain the rationale behind each decision made and share the unexpected findings we encountered during the process.