Recommender Systems: From Collaborative Filtering to Matrix Factorization
Open Netflix and stare at the homepage. Open Spotify and look at Discover Weekly. Open YouTube and watch "Up next" auto-queue. Open Amazon and read "Customers who bought this also bought...". Four different products, four different teams, one identical machine under the hood — a recommender that says "you, specifically, will probably want this". By the end of this lesson you'll know how to build one.
Learning Objectives
After this lesson, you will be able to:
Distinguish content-based filtering from collaborative filtering and know exactly which one you'd reach for in each business situation
Compute user-user and item-item similarities with cosine, Pearson, and adjusted-cosine — and know why item-item dominates production at scale
Understand matrix factorization as low-rank approximation of a sparse user-item matrix — SVD, Funk-SVD, NMF, and implicit-feedback ALS
Diagnose and fix the cold-start problem for new users, new items, and brand-new platforms using content bridges, popularity priors, and side information
Pick the right evaluation metric — RMSE vs Recall@K vs NDCG — and understand why offline metrics almost always overstate online wins
Trace the real production stack used at Netflix, Spotify, and YouTube: candidate generation -> ranking -> reranking, with diversity and freshness constraints
Don't worry if "matrix factorization" sounds intimidating — by the end of this lesson you'll see it's just "split one big sparse table into two smaller dense ones, then multiply them back". That's it. That's the whole idea that won the Netflix Prize.
Every consumer product with a catalog larger than 30 items needs a recommender. The math is brutal: Netflix has ~17,000 titles, Spotify has 100M+ tracks, YouTube uploads 500 hours of video per minute, Amazon has 600M+ products. No human can browse those catalogs. The recommender is the interface — it decides what you see, in what order, on every screen.
There are three broad families of recommenders, and almost every production system blends all three:
Content-based filtering — recommend items similar to ones the user already liked, where "similar" is defined by item features (genre, tempo, color, author, …).
Collaborative filtering — recommend items that similar users liked, ignoring the items' content entirely. "People who watched X also watched Y."
Matrix factorization & embeddings — model both users and items as vectors in the same latent space; recommend by dot product. The dominant modern approach.
We'll build each one up from scratch, then look at how Netflix and Spotify glue them together.
Content-based filtering treats each item as a feature vector and each user as a weighted blend of feature vectors over items they've liked. A new item is recommended if it's geometrically close to the user's profile.
A toy example. A movie has features (is_scifi, is_romance, is_animated, decade=2010s, runtime=120). You rated Inception 5, La La Land 4, and Toy Story 5. Your profile is a weighted average of those feature vectors (heavy on is_scifi and is_animated, moderate on is_romance). A new candidate, Interstellar (heavy on is_scifi, light on everything else), scores high on cosine similarity with your profile. We recommend it.
The good
No cold start for items. A brand-new movie uploaded yesterday has features; we can score it immediately even before a single person rates it. This is huge for news, podcasts, and any catalog that grows hourly.
Interpretable. "We recommended this because you like sci-fi and 2010s movies" is an explanation we can actually show the user.
Per-user models. Each user's profile is independent — no shared training, easy to update online.
The bad
Filter bubble. The model can only recommend things that look like what you've already liked. If your profile screams sci-fi, you will never see the brilliant indie romcom that would have surprised you.
Garbage features = garbage recs. The model is only as good as the item features you engineer. For a song, "danceability" and "valence" are useful; for a t-shirt, defining the feature vector is harder.
New-user cold start. A user with zero ratings has an undefined profile vector.
What Do You Think?
A music streaming startup just launched. They have 100 million tracks (all with rich audio features) but only 1,000 users so far, and most users have liked fewer than 5 songs. Which approach will work better on day 1: content-based filtering or collaborative filtering?
#Collaborative Filtering: "People Like You Liked This"
Collaborative filtering (CF) flips the problem: instead of comparing items by their features, it compares users by their behavior, then borrows the recommendations of behavioral twins. The killer property: CF makes no assumption about what the items are. It works equally well for movies, songs, shoes, podcasts, or news articles — as long as you have a matrix of (user, item, rating) tuples.
Build the user-item matrix R: rows are users, columns are items, entries are ratings (or clicks, or watch time, or any signal). Almost all the entries are missing — that's the whole point. You have 200M users and 17,000 movies; each user has rated maybe 50 movies. R is 99.99% sparse.
CF comes in two flavors depending on which axis you compare along.
For a target user u, find the K most similar users (their "neighbors"). Predict u's rating for item i as a weighted average of what those neighbors rated i.
The classical similarity is Pearson correlation over commonly-rated items:
Treat a user's row in R as a vector in item-space. Cosine similarity between two such vectors is the dot product (after normalization). When we say "users u and v are similar", we mean their rating vectors point in roughly the same direction. The visualization above shows exactly that geometric picture — dot product is compatibility.
Amazon's 2003 paper "Item-to-item collaborative filtering" flipped the axis. Instead of finding similar users (an expensive search across hundreds of millions of users), precompute the item-item similarity matrix: how often do items i and j co-occur in users' rated sets, and with what agreement?
To predict u's rating on item i: look at items u has already rated, weight their ratings by how similar each of those items is to i, and take the weighted average.
r^ui=∑j∈Iu∣sim(i,j)∣∑j∈Iusim(i,j)⋅ruj
#Adjusted Cosine: The Trick That Made Item-Based CF Work
Plain cosine similarity over items has a hidden bug: it doesn't account for user rating bias. User A rates everything 4-5; user B rates everything 1-2. Two items that A rates and B rates look identical by cosine — but really, both users are just expressing baseline behavior, not item similarity.
Adjusted cosine subtracts each user's mean before comparing items:
You're building a recommender for a site with 10 million users and 100,000 items. Each user has touched ~50 items, each item has ~5,000 ratings. Would you use user-based CF or item-based CF for production?
Now let's actually build user-based CF from scratch on a tiny MovieLens-style dataset, compute cosine similarities, and surface top-N recommendations.
Loading visualization...
Quick check
A new user signs up to a music streaming service, listens to nothing, and immediately taps 'For You'. What does *pure* collaborative filtering predict for them?
#Matrix Factorization: The Idea That Won the Netflix Prize
Tune the latent factors and watch two skinny embedding matrices reconstruct the full user-item ratings table.
If R were dense, we could factor it exactly with the singular value decomposition:
R=UΣV⊤≈UkΣkVk⊤=PQ⊤
Loading visualization...
The SVD viz above lets you decompose a matrix and dial down the rank — watch how a rank-2 or rank-5 approximation still captures most of the structure of a much larger matrix. This is exactly what matrix factorization does to the user-item table: throw away the high-rank noise, keep the low-rank "true taste" signal.
#Why Plain SVD Doesn't Work on Real Rec Data: Funk-SVD
There's a catch. SVD as stated requires R to be fully observed. Real R is 99.99% missing. The classical workaround — fill missing entries with the column mean and SVD that — was the standard approach until 2006. It works terribly: you're factoring noise.
In 2006, Simon Funk (an amateur ML hobbyist) posted a blog entry during the Netflix Prize describing what is now called Funk-SVD (sometimes just "regularized matrix factorization"). The idea: don't bother with the missing entries at all. Just optimize P and Q via gradient descent on the observed entries only:
P,Qmin(u,i)∈Ω∑(rui−pu⋅qi)2+λ(∥pu∥2+∥qi∥2)
This single change — only summing over observed entries plus L2 regularization — turned matrix factorization from "academic curiosity" into "production-grade recommender". Funk-SVD shaved 7% off Netflix's RMSE essentially overnight and became the bedrock of the eventual Netflix Prize winner.
NMF adds one constraint to MF: every entry of P and Q must be non-negative.
R≈PQ⊤,Pij≥0,Qij≥0
NMF tends to surface cleaner, more interpretable factors but typically yields slightly worse RMSE than Funk-SVD on rating prediction. Use it when you want to inspect the latent factors (e.g., "what genre is the model implicitly building?").
#Implicit Feedback ALS: The Real Workhorse of Modern Recommenders
Here's the dirty secret of production rec systems: explicit ratings barely exist. Netflix removed star ratings in 2017. Spotify never had them. YouTube has thumbs but most users don't tap them. What we actually have is implicit feedback:
Did the user click on this item? (binary)
Did the user watch / stream / read for ≥30s? (binary, sometimes count)
How many seconds did they spend? (continuous count)
Did they finish? Did they re-watch? Did they share?
Implicit feedback is messier than explicit ratings in three ways:
No negatives. A click means "interested". Not clicking can mean "uninterested" or "never saw it". We can't tell.
Confidence scales with count. Watching a song once might be accidental; watching it 50 times is a strong signal.
Sparsity flips. Instead of 99.99% missing, most user-item cells have some implicit signal (or zero, which we can't trust as "negative").
The standard answer is Implicit ALS (Hu, Koren, Volinsky, 2008) — the paper Spotify, Netflix, and most production rec systems literally implement. Define for each (u, i) cell:
Preference p_ = 1 if any interaction, else 0.
Confidence c_ = 1 + α · count_. More interactions = more confidence in the preference.
Then minimize a weighted squared error over all cells (including the implicit zeros), with confidence as the weight:
P,Qminu,i∑cui(pui−pu⋅qi)2+λ(∥P∥2+∥Q∥2)
This is what powers Spotify's "Discover Weekly" candidate generation. It's what powers Last.fm. It's what powers most "you might like" rails on streaming services.
Now let's actually build matrix factorization end-to-end, evaluate it with Recall@10, and confirm it beats the user-CF baseline.
Pure content-based and pure CF each have failure modes. Production systems blend them. The canonical hybrid recipe:
Compute multiple candidate signals in parallel: content similarity to recently-watched items, CF predictions from a matrix factorization model, popularity priors, demographic-cohort predictions, contextual signals (device, time of day, screen of the app).
Concatenate the scores as features into a learned ranker (gradient-boosted trees or a neural network) that learns the optimal blend per query.
Apply business rules and reranking on top: diversity ("don't show 8 sci-fi movies in a row"), freshness ("show at least 2 items added this week"), exploration ("inject one item from a long-tail genre to learn user signal there").
Netflix has openly described this exact architecture in their tech blog. So have YouTube (the "two-tower" recall + ranking + reranking pipeline) and Spotify (BaRT and the Discover Weekly pipeline).
Popularity baseline. When in doubt, show the items most people like. Boring but rarely catastrophic.
2. New item (zero ratings). CF can't score it because no one has interacted yet.
Content-based bridge. Embed the item from its features (genre, audio embeddings, NLP on the description) and find users whose existing taste vector is similar.
Exploration budget. Deliberately surface new items to a small slice of users to gather ratings fast (multi-armed bandit framing).
Editorial seeding. Humans hand-pick the first few users to expose new content to. Spotify's New Music Friday is partially this.
3. New platform (no data at all). Both fixes above fail simultaneously — you have neither users nor history.
Bootstrap from a related platform. Spotify originally used Last.fm scrobble data. Many startups buy or scrape ratings from public datasets (MovieLens, GoodReads) to seed their model.
Content-only recs until you have data. Pure feature-matching, no CF.
Editorial curation. Humans pick the homepage. This is what Apple Music does for new genres.
Quick check
A streaming service is about to launch in a new country with a fresh user base. What's the *most defensible* cold-start strategy for the first 4-6 weeks?
We've touched this already; let's nail it down because the choice changes everything downstream.
Aspect
Explicit (ratings)
Implicit (clicks/streams)
Source
User actively rates (1-5 stars, thumbs)
User behavior (clicks, watch time, scrolls)
Volume
Sparse — most users never rate
Dense — everyone clicks
Bias
Heavy selection bias (people rate movies they love or hate, ignore the middle)
Different bias (popular items get more clicks regardless of quality)
Negatives
Real (1-star = explicit dislike)
Inferred — absence of a click is not a dislike
Confidence
Constant per rating
Scales with count (50 listens > 1 listen)
Standard model
Funk-SVD / SVD++
Implicit ALS, BPR (Bayesian Personalized Ranking)
Quick check
You're training a recommender on YouTube watch data. For each (user, video) pair you have either a 'click' (user opened the video) or nothing. Should you treat 'no click' as a negative (preference = 0)?
Recommender evaluation is uniquely treacherous. Here's the hierarchy of metrics, from least to most aligned with what users actually experience:
Prediction-accuracy metrics (least aligned):
RMSE / MAE. Root mean squared error / mean absolute error on the predicted rating. Used heavily in the academic literature and the Netflix Prize era. Penalizes prediction error, ignores ranking.
Ranking metrics (better):
Hit Rate@K. For each held-out user-item pair, was the held-out item in the model's top-K recommendations? Binary, easy to interpret.
Recall@K. Fraction of relevant items that appeared in the top K. Maps directly to "did we surface the good stuff?"
Precision@K. Fraction of top-K items that were relevant.
MAP (Mean Average Precision). Averages precision over all the positions where a relevant item appeared. Rewards putting relevant items near the top.
NDCG (Normalized Discounted Cumulative Gain). Log-discounted ranking score that handles graded relevance. The gold standard for ranking quality.
Beyond accuracy:
Diversity. Average pairwise dissimilarity of the recommended set. Are we showing 10 nearly-identical items?
Novelty. How unfamiliar are the recommendations to the user? Recommending only re-watches has zero novelty.
Serendipity. Recommendations that are useful and surprising. Hard to measure directly; usually a downstream effect of mixing exploration into the ranking.
Coverage. What fraction of the catalog ever gets recommended? A model that only ever recommends the top 100 popular items has terrible coverage and starves the long tail.
The treachery: offline metrics almost always overestimate online performance, sometimes catastrophically. Three reasons:
Selection bias in the test set. The items you have ratings for are the items users chose to engage with — already biased toward popularity and the existing recommender's choices. A model that does well predicting those ratings might do badly on items the system would have otherwise surfaced.
No counterfactual. Offline you measure "given users actually clicked X, did the model predict X?" — but the production recommender's job is to find X, not predict it after the fact. The set of items the user would have clicked if we'd surfaced them is unknowable from logs.
Position bias. Click data is overwhelmingly biased toward whatever was shown at the top. A new model that puts the same item lower will appear to "lose clicks" offline even if it would have won them online.
The industry-standard mitigation is A/B testing every recommender change in production, treating offline metrics as a gate (must clear a minimum bar) rather than a decision (which model wins). Pinterest, Netflix, Spotify, YouTube all openly describe this practice. The most respected paper on this is "Offline A/B testing for Recommender Systems" (Gilotte et al., 2018).
A perfect-accuracy recommender that shows you 10 sci-fi thrillers is a bad product. Users get bored. They want variety, occasional surprise, and a sense the system "knows them" without being a mirror.
Three techniques used in production reranking:
Maximal Marginal Relevance (MMR). After scoring candidates, iteratively pick the next item that maximizes λ · relevance - (1-λ) · max_similarity_to_already_picked. This explicitly trades off relevance against redundancy.
Determinantal Point Processes (DPPs). A probabilistic framework where the probability of selecting a set of items is proportional to the determinant of their feature similarity matrix. Diverse sets get higher determinants. Used by Hulu and Netflix.
Exploration bonuses. Multi-armed-bandit style: occasionally boost the score of low-confidence items to gather signal. Spotify's "Smart Shuffle" is partially this.
#Modern Direction: Two-Tower Neural Recs and Sequential Models
Matrix factorization is the bedrock, but the frontier moved past it around 2018.
Two-tower architectures generalize MF: a user tower (any deep network) produces a user embedding from user features and history; an item tower produces an item embedding from item features. Train both with the dot product as the score. The "factor" matrices P and Q are now learned functions of features, which lets the same model serve content-based, CF, and contextual recommendations under one objective.
YouTube's two-tower recall model retrieves ~hundreds of candidates per query from a billion-item index using approximate nearest neighbor (FAISS / ScaNN) on item embeddings. Their published architecture is the canonical reference (Covington et al., 2016; further evolved in their RecSys 2019 paper).
Pinterest's Pinnability and Pinformer use similar dual-encoder structures.
Sequential / session-aware models treat a user's interaction history as an ordered sequence rather than a bag. They're transformers, basically, applied to (item, position) pairs.
SASRec (Kang & McAuley, 2018) uses self-attention over the user's history to predict the next item.
BERT4Rec (Sun et al., 2019) trains a masked-item model in the same spirit as BERT — randomly mask interactions, predict them from surrounding context.
These are essential when order matters (you binge a series in order; you finish one song before the next on a playlist).
We'll go deeper on these in the Transformers track, where SASRec and BERT4Rec slot in as direct applications of self-attention. For now, the important takeaway: sequence-aware models routinely beat static MF on session-heavy data (music streaming, video, news feed) by 5-15% on NDCG. They do not beat MF on rating prediction with no temporal signal — there, MF is still the right hammer.
#The Production Stack: Candidate Generation -> Ranking -> Reranking
The actual machine on a Netflix homepage is a pipeline, not a model:
Stage 1: Candidate generation (recall). Given a user, retrieve ~1,000 candidate items from a catalog of ~17,000 (Netflix) or ~100M (Spotify). Multiple parallel recall sources:
The recall stage optimizes for recall@1000, not precision. We just need the right item somewhere in our 1,000-candidate set.
Stage 2: Ranking. A heavier model (gradient-boosted tree or DNN) scores each of the 1,000 candidates with many features: predicted watch probability, watch-time-given-click, freshness, user-specific signals (time of day, device, last 5 actions). This stage optimizes for NDCG / expected user engagement.
Stage 3: Reranking. Apply diversity (MMR), freshness, business rules ("don't show kids' content in adult profile"), exploration ("inject 1 long-tail item for signal"), and editorial overrides. This stage optimizes for user satisfaction across the page, not just per-item relevance.
The three stages use different models and different metrics deliberately — early stages are cheap and high-recall, later stages are expensive and high-precision. The same architecture is used at Pinterest, YouTube, TikTok, Amazon, and Spotify.
Three recommender families, blended in production: content-based (item features), collaborative filtering (user behavior), and matrix factorization / embeddings (latent factors). Pure approaches lose to hybrids that combine all three.
Item-based CF dominated user-based CF in production because item-item similarity is stable, precomputable offline, and reduces online prediction to a small weighted sum over the user's history.
Matrix factorization is the soul of modern recs: decompose the sparse user-item matrix into low-rank user and item embeddings, predict via dot product. Funk-SVD (regularized MF on observed entries) was the Netflix Prize breakthrough; Implicit ALS handles the implicit-feedback case that dominates real products.
Cold start has three flavors, each with a different fix: new user (onboarding survey + popularity), new item (content bridge + exploration), new platform (bootstrap from another dataset or editorial curation).
RMSE is a trap when the product is ranking. Optimize Recall@K, NDCG, and MAP instead, and always validate online with A/B tests because offline metrics overestimate gains.
Beyond accuracy matters: diversity, novelty, serendipity, and coverage are the difference between a model that looks good in a paper and one users actually love.
Why did matrix factorization with Funk-SVD beat the 'fill missing entries with the column mean, then SVD' approach so dramatically on Netflix data?
You can now build a recommender from first principles, factor a sparse user-item matrix, evaluate it with the right metrics, and reason about why your offline gains might not survive contact with production. Next up: anomaly detection — what happens when you flip recommenders inside out and ask "which points don't fit?" instead of "which are most similar?"
Standard metric
RMSE
Recall@K, NDCG, AUC
Pure CF can't help; engineer signal at signup
You're a real product team
Multi-stage stack: candidate gen → ranking → reranking, with A/B tests
This is what every production system actually does