Recommender Systems, Part 2: Implicit Feedback, Cold Start, Evaluation and the Production Stack
Part 1 built the models. A model that scores well on a spreadsheet is still not a product. Real users arrive with no history, they click instead of rating, the score you measured offline turns out to flatter you, and the homepage has to be assembled in milliseconds. This part covers everything between a working model and a recommender people actually use.
Learning Objectives
After this lesson, you will be able to:
Explain implicit-feedback ALS (preference and confidence) and what NMF's non-negativity constraint buys
Explain why production systems blend content-based, collaborative and factorization models, and what a learned ranker adds
Diagnose the three kinds of cold start (new user, new item, new platform) and pick a fix for each
Choose models and metrics for explicit versus implicit feedback, and say why a missing click is not a dislike
Compute Recall@K and NDCG by hand, and explain why offline gains overstate online ones
Trace candidate generation, ranking and reranking, and say what each stage optimizes
In plain words: Part 1 was about guessing well. Part 2 is about guessing usefully. That means having an answer on the first day, measuring success the way the user feels it, and showing a page that is varied and not only accurate. None of it needs new maths. It needs a clear head about what the model is for.
Words you need10 terms
Non-negative matrix factorization. Every factor value must be zero or above, so each item is an additive blend of themes.
The moment a user, an item or a whole platform has no history yet, so the ratings table has nothing to say about it.
Behaviour that shows interest without anyone saying so: clicks, plays, minutes watched. It is plentiful, but a missing click does not mean a dislike.
How much weight one observed cell gets. Fifty listens earn more confidence than one.
Of the items the user really liked, the share that appear in the top K of the list.
A ranking score that gives more credit when liked items sit higher in the list. 1 is a perfect order.
The first stage, which cheaply pulls a shortlist of about a thousand items out of the whole catalog.
The last stage, which reorders the ranked list for variety, freshness and business rules.
Showing two versions to different random groups of real users and comparing what they do.
How different the items in one list are from each other. Ten near-identical films have low diversity.
Part 1 ended with Funk-SVD on explicit star ratings. Two variants matter in practice. NMF changes what the factors mean, and implicit ALS changes what the data means. The second is the one that runs most real products.
NMF adds one constraint to MF: every entry of P and Q must be non-negative.
R≈PQ⊤,Pij≥0,Qij≥0
NMF tends to surface cleaner, more interpretable factors but typically yields slightly worse RMSE than Funk-SVD on rating prediction. Use it when you want to inspect the latent factors (e.g., "what genre is the model implicitly building?").
#Implicit Feedback ALS: The Real Workhorse of Modern Recommenders
Here's the dirty secret of production rec systems: explicit ratings barely exist. Netflix removed star ratings in 2017. Spotify never had them. YouTube has thumbs but most users don't tap them. What we actually have is implicit feedback:
Did the user click on this item? (binary)
Did the user watch / stream / read for ≥30s? (binary, sometimes count)
How many seconds did they spend? (continuous count)
Did they finish? Did they re-watch? Did they share?
Implicit feedback is messier than explicit ratings in three ways:
No negatives. A click means "interested". Not clicking can mean "uninterested" or "never saw it". We can't tell.
Confidence scales with count. Watching a song once might be accidental; watching it 50 times is a strong signal.
Sparsity flips. Instead of 99.99% missing, most user-item cells have some implicit signal (or zero, which we can't trust as "negative").
The standard answer is Implicit ALS (Hu, Koren, Volinsky, 2008) — the paper Spotify, Netflix, and most production rec systems literally implement. Define for each (u, i) cell:
Preference p_ = 1 if any interaction, else 0.
Confidence c_ = 1 + α · count_. More interactions = more confidence in the preference.
Then minimize a weighted squared error over all cells (including the implicit zeros), with confidence as the weight:
P,Qminu,i∑cui(pui−pu⋅qi)2+λ(∥P∥2+∥Q∥2)
This is what powers Spotify's "Discover Weekly" candidate generation. It's what powers Last.fm. It's what powers most "you might like" rails on streaming services.
The implicit-feedback step has the same shape as the explicit one, with a weight on every cell. To avoid a clash between the preference p_ui and the user vector p_u, the derivation writes the preference vector of user u as pi_u.
Now let's actually build matrix factorization end-to-end, evaluate it with Recall@10, and confirm it beats the user-CF baseline.
Pure content-based and pure CF each have failure modes. Production systems blend them. The canonical hybrid recipe:
Compute multiple candidate signals in parallel: content similarity to recently-watched items, CF predictions from a matrix factorization model, popularity priors, demographic-cohort predictions, contextual signals (device, time of day, screen of the app).
Concatenate the scores as features into a learned ranker (gradient-boosted trees or a neural network) that learns the optimal blend per query.
Apply business rules and reranking on top: diversity ("don't show 8 sci-fi movies in a row"), freshness ("show at least 2 items added this week"), exploration ("inject one item from a long-tail genre to learn user signal there").
Netflix has openly described this exact architecture in their tech blog. So have YouTube (the "two-tower" recall + ranking + reranking pipeline) and Spotify (BaRT and the Discover Weekly pipeline).
Popularity baseline. When in doubt, show the items most people like. Boring but rarely catastrophic.
2. New item (zero ratings). CF can't score it because no one has interacted yet.
Content-based bridge. Embed the item from its features (genre, audio embeddings, NLP on the description) and find users whose existing taste vector is similar.
Exploration budget. Deliberately surface new items to a small slice of users to gather ratings fast (multi-armed bandit framing).
Editorial seeding. Humans hand-pick the first few users to expose new content to. Spotify's New Music Friday is partially this.
3. New platform (no data at all). Both fixes above fail simultaneously — you have neither users nor history.
Bootstrap from a related platform. Spotify originally used Last.fm scrobble data. Many startups buy or scrape ratings from public datasets (MovieLens, GoodReads) to seed their model.
Content-only recs until you have data. Pure feature-matching, no CF.
Editorial curation. Humans pick the homepage. This is what Apple Music does for new genres.
Quick check
A streaming service is about to launch in a new country with a fresh user base. What's the *most defensible* cold-start strategy for the first 4-6 weeks?
We've touched this already; let's nail it down because the choice changes everything downstream.
Aspect
Explicit (ratings)
Implicit (clicks/streams)
Source
User actively rates (1-5 stars, thumbs)
User behavior (clicks, watch time, scrolls)
Volume
Sparse — most users never rate
Dense — everyone clicks
Bias
Heavy selection bias (people rate movies they love or hate, ignore the middle)
Different bias (popular items get more clicks regardless of quality)
Negatives
Real (1-star = explicit dislike)
Inferred — absence of a click is not a dislike
Confidence
Constant per rating
Scales with count (50 listens > 1 listen)
Standard model
Funk-SVD / SVD++
Implicit ALS, BPR (Bayesian Personalized Ranking)
Quick check
You're training a recommender on YouTube watch data. For each (user, video) pair you have either a 'click' (user opened the video) or nothing. Should you treat 'no click' as a negative (preference = 0)?
Recommender evaluation is uniquely treacherous. Here's the hierarchy of metrics, from least to most aligned with what users actually experience:
Prediction-accuracy metrics (least aligned):
RMSE / MAE. Root mean squared error / mean absolute error on the predicted rating. Used heavily in the academic literature and the Netflix Prize era. Penalizes prediction error, ignores ranking.
Ranking metrics (better):
Hit Rate@K. For each held-out user-item pair, was the held-out item in the model's top-K recommendations? Binary, easy to interpret.
Recall@K. Fraction of relevant items that appeared in the top K. Maps directly to "did we surface the good stuff?"
Precision@K. Fraction of top-K items that were relevant.
MAP (Mean Average Precision). Averages precision over all the positions where a relevant item appeared. Rewards putting relevant items near the top.
NDCG (Normalized Discounted Cumulative Gain). Log-discounted ranking score that handles graded relevance. The gold standard for ranking quality.
Beyond accuracy:
Diversity. Average pairwise dissimilarity of the recommended set. Are we showing 10 nearly-identical items?
Novelty. How unfamiliar are the recommendations to the user? Recommending only re-watches has zero novelty.
Serendipity. Recommendations that are useful and surprising. Hard to measure directly; usually a downstream effect of mixing exploration into the ranking.
Coverage. What fraction of the catalog ever gets recommended? A model that only ever recommends the top 100 popular items has terrible coverage and starves the long tail.
A tiny worked example makes these concrete. A user liked three held-out items. The model's list puts two of them at ranks 1 and 3, and the third falls at rank 6, outside the top five. Hit Rate@5 is 1, because at least one liked item made the top five. Recall@5 is 2 out of 3. Precision@5 is 2 out of 5, which is 0.4.
NDCG also rewards position. A liked item at rank r earns 1 divided by log2(r + 1), so rank 1 earns 1.00 and rank 3 earns 0.50. The list scores 1.50, and a perfect list with the three liked items at ranks 1, 2 and 3 would score 1 + 0.63 + 0.50 = 2.13. So NDCG@5 is 1.50 / 2.13 = 0.70. Slide the rank-3 item down to rank 5 and Recall@5 stays at 2 out of 3, but its credit falls to 0.39 and NDCG@5 falls to 0.65. Recall cannot see the difference. NDCG can.
The treachery: offline metrics almost always overestimate online performance, sometimes catastrophically. Three reasons:
Selection bias in the test set. The items you have ratings for are the items users chose to engage with — already biased toward popularity and the existing recommender's choices. A model that does well predicting those ratings might do badly on items the system would have otherwise surfaced.
No counterfactual. Offline you measure "given users actually clicked X, did the model predict X?", but the production recommender's job is to find X, not predict it after the fact. The set of items the user would have clicked if we'd surfaced them is unknowable from logs.
Position bias. Click data is overwhelmingly biased toward whatever was shown at the top. A new model that puts the same item lower will appear to "lose clicks" offline even if it would have won them online.
The industry-standard mitigation is A/B testing every recommender change in production, treating offline metrics as a gate (must clear a minimum bar) rather than a decision (which model wins). Pinterest, Netflix, Spotify, YouTube all openly describe this practice. The most respected paper on this is "Offline A/B testing for Recommender Systems" (Gilotte et al., 2018).
A perfect-accuracy recommender that shows you 10 sci-fi thrillers is a bad product. Users get bored. They want variety, occasional surprise, and a sense the system "knows them" without being a mirror.
Three techniques used in production reranking:
Maximal Marginal Relevance (MMR). After scoring candidates, iteratively pick the next item that maximizes λ · relevance - (1-λ) · max_similarity_to_already_picked. This explicitly trades off relevance against redundancy.
Determinantal Point Processes (DPPs). A probabilistic framework where the probability of selecting a set of items is proportional to the determinant of their feature similarity matrix. Diverse sets get higher determinants. Used by Hulu and Netflix.
Exploration bonuses. Multi-armed-bandit style: occasionally boost the score of low-confidence items to gather signal. Spotify's "Smart Shuffle" is partially this.
A small example shows the trade. Take lambda = 0.7. Candidate A has relevance 0.9, but a thriller already in the slate is 0.95 similar to it. Candidate B has relevance 0.7 and is at most 0.2 similar to anything chosen so far. The scores are 0.7 x 0.9 - 0.3 x 0.95 = 0.345 for A and 0.7 x 0.7 - 0.3 x 0.2 = 0.43 for B. B goes in next, even though A is the more relevant item.
#Modern Direction: Two-Tower Neural Recs and Sequential Models
Matrix factorization is the bedrock, but the frontier moved past it around 2018.
Two-tower architectures generalize MF: a user tower (any deep network) produces a user embedding from user features and history; an item tower produces an item embedding from item features. Train both with the dot product as the score. The "factor" matrices P and Q are now learned functions of features, which lets the same model serve content-based, CF, and contextual recommendations under one objective.
YouTube's two-tower recall model retrieves ~hundreds of candidates per query from a billion-item index using approximate nearest neighbor (FAISS / ScaNN) on item embeddings. Their published architecture is the canonical reference (Covington et al., 2016; further evolved in their RecSys 2019 paper).
Pinterest's Pinnability and Pinformer use similar dual-encoder structures.
Sequential / session-aware models treat a user's interaction history as an ordered sequence rather than a bag. They're transformers, basically, applied to (item, position) pairs.
SASRec (Kang & McAuley, 2018) uses self-attention over the user's history to predict the next item.
BERT4Rec (Sun et al., 2019) trains a masked-item model in the same spirit as BERT — randomly mask interactions, predict them from surrounding context.
These are essential when order matters (you binge a series in order; you finish one song before the next on a playlist).
We'll go deeper on these in the Transformers track, where SASRec and BERT4Rec slot in as direct applications of self-attention. For now, the important takeaway: sequence-aware models routinely beat static MF on session-heavy data (music streaming, video, news feed) by 5-15% on NDCG. They do not beat MF on rating prediction with no temporal signal — there, MF is still the right hammer.
#The Production Stack: Candidate Generation -> Ranking -> Reranking
The actual machine on a Netflix homepage is a pipeline, not a model:
Stage 1: Candidate generation (recall). Given a user, retrieve ~1,000 candidate items from a catalog of ~17,000 (Netflix) or ~100M (Spotify). Multiple parallel recall sources:
The recall stage optimizes for recall@1000, not precision. We just need the right item somewhere in our 1,000-candidate set.
Stage 2: Ranking. A heavier model (gradient-boosted tree or DNN) scores each of the 1,000 candidates with many features: predicted watch probability, watch-time-given-click, freshness, user-specific signals (time of day, device, last 5 actions). This stage optimizes for NDCG / expected user engagement.
Stage 3: Reranking. Apply diversity (MMR), freshness, business rules ("don't show kids' content in adult profile"), exploration ("inject 1 long-tail item for signal"), and editorial overrides. This stage optimizes for user satisfaction across the page, not just per-item relevance.
The three stages use different models and different metrics deliberately — early stages are cheap and high-recall, later stages are expensive and high-precision. Variations of this architecture are common across large platforms.
Real products run on implicit feedback. Implicit ALS turns clicks and plays into a preference plus a confidence, treats unobserved cells as weak evidence and not as dislikes, and solves each user and item vector in closed form
Production systems blend. Content-based scores, collaborative scores, factorization scores and popularity priors become features for a learned ranker, so the matrix factorizer is one input to something bigger rather than the winner
Cold start has three flavors, each with a different fix: new user (onboarding survey + popularity), new item (content bridge + exploration), new platform (bootstrap from another dataset or editorial curation).
RMSE is a trap when the product is ranking. Optimize Recall@K, NDCG, and MAP instead, and always validate online with A/B tests because offline metrics overestimate gains.
Beyond accuracy matters: diversity, novelty, serendipity, and coverage are the difference between a model that looks good in a paper and one users actually love.
A homepage is a pipeline of three stages. Candidate generation is cheap and judged on recall, ranking is heavier and judged on NDCG, and reranking handles variety, freshness and business rules
A streaming service A/B tests two recommenders. Model A has 8% better offline NDCG than Model B. In a 2-week production A/B test, Model A has *worse* completion rate. What's the most likely explanation?
You can now build a recommender from first principles, factor a sparse user-item matrix, evaluate it with the right metrics, and reason about why your offline gains might not survive contact with production. Next up: causal inference — the difference between "users who saw this recommendation bought more" and "this recommendation caused them to buy more."
Standard metric
RMSE
Recall@K, NDCG, AUC
Pure CF can't help; engineer signal at signup
You're a real product team
Multi-stage stack: candidate gen → ranking → reranking, with A/B tests
This is what every production system actually does