Thursday, November 12, 2026

Building a Multi-Stage Recommender System: Candidate Generation, Ranking and Re-Ranking

TL;DR

A multi-stage recommender narrows a big catalog in steps: cheap candidate generation, a richer ranking model, then re-ranking rules for the final list. This guide builds all three in Python on a synthetic catalog and measures what each stage adds. Like any good audition process, not every contestant needs to reach the final round.

The short version, in three stops:

Say an online store has 10,000 products and wants to show each visitor the 10 they are most likely to click. A strong model could score every product for every visitor, but it would have to do that in a fraction of a second, millions of times a day. Most of that work would go into products nobody would ever show.

A multi-stage recommender system splits the job. Candidate generation quickly narrows the full catalog to a few hundred items that are plausibly relevant. Ranking scores that short list with a richer model. Re-ranking applies business rules, such as diversity or filtering, to produce the final list. This is the design YouTube described in its 2016 paper on deep neural networks for recommendations, and it remains the standard shape of large recommenders. This guide builds all three stages in Python on a synthetic catalog and shows what each stage contributes, with real output.

Key Takeaways

  • A multi-stage recommender uses a cheap model to find candidates, a more expensive model to rank them, and rules to re-rank the final list.
  • Candidate generation is judged on recall: does the short list contain the items the user would like? Ranking is judged on the order of the top few.
  • In our toy system, ranking 322 candidates gave the same quality as ranking all 10,000 items (47.5% vs 47.8% expected click rate), at about 3% of the scoring work.
  • Re-ranking for diversity has a cost. Capping each category at two items raised the categories per list from 2.1 to 5.7 and lowered the expected click rate from 47.5% to 40.9%.
  • Measure each stage on its own. A weak candidate stage caps what the ranker can do, no matter how good the ranker is.

What Is a Multi-Stage Recommender System?

A multi-stage recommender system is a pipeline that filters a large catalog down to a short list in steps, using faster and simpler methods early and slower, more accurate methods late. Each stage works on far fewer items than the one before it, so the expensive model only runs where it matters.

The best-known public description is Deep Neural Networks for YouTube Recommendations by Paul Covington, Jay Adams and Emre Sargin (RecSys 2016). It splits the system into a candidate generation network, which selects a few hundred videos from a corpus of millions, and a ranking network, which scores those candidates with many more features. Meta has described Instagram Explore as a similar multi-stage funnel in Powered by AI: Instagram’s Explore Recommender System, with candidate sourcing followed by ranking passes.

Stage Input Output Goal Typical methods
Candidate generation Full catalog Hundreds to thousands of items High recall, very fast Embedding nearest neighbors, co-occurrence, popularity, rules
Ranking Candidates Scored list Accurate ordering Gradient boosted trees, deep networks with many features
Re-ranking Top of ranked list Final list Business and user experience goals Diversity caps, filters, freshness boosts, de-duplication

The Toy Catalog

To see what each stage does, we need data where we know the right answer. The script creates 5,000 users and 10,000 items. Each item belongs to one of 20 categories, and items in the same category have similar hidden traits. Each user has hidden tastes, and the chance that a user clicks an item depends on how well the two match plus an overall item quality score. Every user has already seen 400 random items, and their clicks form the history the system learns from.

Because the click probabilities are known, we can compute the expected click rate of any list exactly. That is the “expected CTR” in the results below. In a real system you would estimate it with offline metrics and then an A/B test.

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
N_USERS, N_ITEMS, DIM, N_CATS = 5_000, 10_000, 8, 20

# ---- A synthetic catalog with hidden tastes ----------------------------
user_taste = rng.normal(size=(N_USERS, DIM))
item_cat = rng.integers(0, N_CATS, N_ITEMS)
cat_center = rng.normal(size=(N_CATS, DIM))
item_taste = 0.8 * cat_center[item_cat] + 0.6 * rng.normal(size=(N_ITEMS, DIM))
item_quality = rng.normal(size=N_ITEMS)           # some items are just better
true_affinity = user_taste @ item_taste.T / np.sqrt(DIM) + 0.5 * item_quality


def click_prob(u, items):
    return 1 / (1 + np.exp(-(true_affinity[u, items] - 2.5)))


# Each user has seen 400 random items and clicked some of them
history = np.zeros((N_USERS, N_ITEMS), dtype=np.float32)
seen = rng.integers(0, N_ITEMS, (N_USERS, 400))
for u in range(N_USERS):
    clicks = rng.random(400) < click_prob(u, seen[u])
    history[u, seen[u][clicks]] = 1.0
popularity = history.sum(axis=0)

Stage 1: Candidate Generation

Candidate generation has one job: keep the items a user might like and drop the rest, quickly. It does not need to order them well. A common approach is to learn an embedding (a short vector) for every item and every user, then retrieve the items whose vectors are closest to the user’s.

Here, a truncated SVD of the click history gives an 8-number embedding for each item. A user’s embedding is the sum of the embeddings of items they clicked. The stage takes the 300 items with the highest dot product, adds the 50 most popular items as a second source, and removes items the user already clicked.

# ---- Stage 1: candidate generation (cheap, recall-oriented) ------------
svd = TruncatedSVD(n_components=DIM, random_state=0).fit(history)
item_emb = svd.components_.T                       # (N_ITEMS, DIM)
user_emb = history @ item_emb                      # (N_USERS, DIM)


def candidates(u, k_emb=300, k_pop=50):
    scores = item_emb @ user_emb[u]
    emb_top = np.argpartition(-scores, k_emb)[:k_emb]
    pop_top = np.argpartition(-popularity, k_pop)[:k_pop]
    cand = np.union1d(emb_top, pop_top)
    return cand[history[u, cand] == 0]             # drop items already clicked

Mixing sources is normal in production. Embedding retrieval finds personal matches, popularity covers new users with little history, and other sources (recently viewed, same brand, editorial picks) fill other gaps. At large scale the dot product search runs in an approximate nearest neighbor index rather than over every item. The same retrieval idea powers search systems, as our guides on hybrid search for retrieval and the BM25S ranking algorithm show.

Stage 2: Ranking

The ranker only sees a few hundred items per user, so it can afford more features and a heavier model. In this toy it uses three features: the embedding score from stage 1, the item’s log popularity, and how often the user clicked items in the same category. A logistic regression learns how to weigh them from logged impressions: 50 random items shown to each of the first 1,000 users, with simulated clicks.

# ---- Stage 2: ranking (richer features, trained on logged impressions) --
def features(u, items):
    emb_score = item_emb[items] @ user_emb[u]
    user_cats = np.bincount(item_cat[history[u] > 0], minlength=N_CATS)
    cat_affinity = user_cats[item_cat[items]] / max(1, user_cats.sum())
    return np.column_stack([emb_score, np.log1p(popularity[items]), cat_affinity])


X_train, y_train = [], []
for u in range(1_000):                             # logged impressions from the first 1,000 users
    shown = rng.integers(0, N_ITEMS, 50)
    X_train.append(features(u, shown))
    y_train.append(rng.random(50) < click_prob(u, shown))
ranker = LogisticRegression().fit(np.vstack(X_train), np.concatenate(y_train))

Real rankers use hundreds of features (user context, time of day, item age, past interactions with similar items) and models such as gradient boosted trees or deep networks. The YouTube paper, for example, trains its ranking network to predict expected watch time rather than clicks, so that click-bait videos do not win. Choosing what the ranker predicts is one of the most important decisions in the system.

Stage 3: Re-Ranking

A list sorted only by predicted clicks tends to repeat itself. If a user likes one category, the top 10 can all come from it. Re-ranking applies rules that the ranker does not see. Here the rule is simple: at most two items per category.

# ---- Stage 3: re-ranking (business rules on a short list) --------------
def rerank(items, scores, k=10, max_per_cat=2):
    out, per_cat = [], {}
    for i in items[np.argsort(-scores)]:
        c = item_cat[i]
        if per_cat.get(c, 0) < max_per_cat:
            out.append(i)
            per_cat[c] = per_cat.get(c, 0) + 1
        if len(out) == k:
            break
    return np.array(out)


def recommend(u):
    cand = candidates(u)
    scores = ranker.predict_proba(features(u, cand))[:, 1]
    return cand, rerank(cand, scores)

Other common re-ranking rules include removing out-of-stock items, limiting how often one seller appears, boosting new items, and spacing out sponsored placements.

Measuring Each Stage

The evaluation runs on 200 users the ranker never trained on (users 1,000 to 1,199). For each one it records how many of the user’s true top 100 items (among those not yet clicked) made it into the candidates, and the expected click rate of several top-10 lists.

# ---- Evaluate on users the ranker never saw ----------------------------
test_users = range(1_000, 1_200)
recall, ctr_final, ctr_random, ctr_ranked, ctr_full, ctr_best = [], [], [], [], [], []
cand_sizes, cats_per_list, cats_ranked = [], [], []
for u in test_users:
    cand, final = recommend(u)
    unseen = np.flatnonzero(history[u] == 0)
    best100 = unseen[np.argsort(-true_affinity[u, unseen])[:100]]
    recall.append(np.isin(best100, cand).mean())
    cand_sizes.append(len(cand))
    ctr_final.append(click_prob(u, final).mean())
    ctr_random.append(click_prob(u, rng.choice(unseen, 10, replace=False)).mean())
    s = ranker.predict_proba(features(u, cand))[:, 1]
    top_ranked = cand[np.argsort(-s)[:10]]                               # no re-ranking
    ctr_ranked.append(click_prob(u, top_ranked).mean())
    cats_ranked.append(len(set(item_cat[top_ranked])))
    s_all = ranker.predict_proba(features(u, unseen))[:, 1]
    ctr_full.append(click_prob(u, unseen[np.argsort(-s_all)[:10]]).mean())  # rank everything
    ctr_best.append(click_prob(u, best100[:10]).mean())                  # oracle
    cats_per_list.append(len(set(item_cat[final])))

print(f"Catalog size:                    {N_ITEMS:,}")
print(f"Avg candidates after stage 1:    {np.mean(cand_sizes):.0f}")
print(f"Recall of each user's true top 100 in candidates: {np.mean(recall):.1%}")
print(f"Ranker weights (emb, log pop, cat): {np.round(ranker.coef_[0], 3)}")
print(f"Expected CTR, random 10 items:   {np.mean(ctr_random):.1%}")
print(f"Expected CTR, ranker on all items: {np.mean(ctr_full):.1%}")
print(f"Expected CTR, ranked candidates: {np.mean(ctr_ranked):.1%}")
print(f"Expected CTR, final 10 items:    {np.mean(ctr_final):.1%}")
print(f"Expected CTR, true best 10:      {np.mean(ctr_best):.1%}")
print(f"Distinct categories, ranked only: {np.mean(cats_ranked):.1f} of 10")
print(f"Distinct categories, re-ranked:  {np.mean(cats_per_list):.1f} of 10")

Output (Python 3.13, NumPy 2.5, scikit-learn 1.9; the full script ran in about 6 seconds in our environment):

Catalog size:                    10,000
Avg candidates after stage 1:    322
Recall of each user's true top 100 in candidates: 36.1%
Ranker weights (emb, log pop, cat): [5.687 0.907 8.07 ]
Expected CTR, random 10 items:   11.0%
Expected CTR, ranker on all items: 47.8%
Expected CTR, ranked candidates: 47.5%
Expected CTR, final 10 items:    40.9%
Expected CTR, true best 10:      69.6%
Distinct categories, ranked only: 2.1 of 10
Distinct categories, re-ranked:  5.7 of 10

What the Results Show

  • Candidate generation keeps the right items cheaply. Stage 1 keeps 322 items, about 3% of the catalog, but 36% of each user’s true top 100. A random 3% would keep about 3%. The ranker then scores 322 items instead of 10,000.
  • The short list costs almost nothing. Ranking only the candidates gives an expected click rate of 47.5%. Ranking every item the user has not clicked, close to 10,000 of them, gives 47.8%. For a 31-fold cut in scoring work, the loss is 0.3 points.
  • Ranking does most of the lifting. A random top 10 has an expected click rate of 11.0%. The ranked list more than quadruples it. The ranker learned to rely most on category affinity and the embedding score, which matches how the data was built.
  • Diversity is a trade-off, not a free win. Without re-ranking, a typical list draws on only 2.1 categories. The two-per-category cap raises that to 5.7 and lowers expected clicks to 40.9%. Whether that trade is worth it depends on goals the click model does not capture, such as long-term satisfaction or discovery. A/B tests are how teams decide.
  • There is room to improve. The true best 10 would reach 69.6%. Most of the gap comes from the simple features and the 8-number embeddings, which is where real systems invest.

Comparing against simple baselines like the random list is good practice at every stage. Our guide to the dummy classifier covers the same idea for classification models.

Going From Toy to Production

  • Train retrieval and ranking separately. They have different goals (recall versus ordering) and different data sizes.
  • Use an approximate nearest neighbor index for candidate generation once the catalog reaches millions of items.
  • Log what you showed. The ranker is trained on impressions, so record every item shown, its position and the outcome. Items never shown never get labels, which can trap a system in its own past choices.
  • Watch latency per stage. Each stage gets a time budget. If ranking takes too long, shrink the candidate set or simplify the features.

Back to the 10,000 Products

The store from the opening does not need to score every product for every visitor. In the toy version, a cheap first stage cut 10,000 products to about 320, a ranker ordered them almost as well as if it had seen the whole catalog, and a simple cap per category spread the final 10 across more categories. Each stage can now be measured and improved on its own, which is the real reason this design has lasted.

Frequently Asked Questions

How Many Candidates Should Stage One Return?

Enough that the items the user would choose are usually in the set, and few enough that the ranker meets its time budget. Public descriptions such as the YouTube paper speak of hundreds of candidates from millions of items. Measure recall at several sizes and stop where gains flatten out.

What Is the Difference Between Ranking and Re-Ranking?

Ranking scores each item on its own with a learned model, usually predicting a click, watch or purchase. Re-ranking looks at the list as a whole and applies goals the model does not capture, such as diversity, freshness or business rules. Re-ranking is often rules, but it can also be a learned model.

Do I Need Deep Learning for a Multi-Stage Recommender?

No. As the toy example shows, matrix factorization for retrieval and logistic regression or gradient boosted trees for ranking already form a working system. Deep models help when you have large amounts of interaction data and many feature types, such as text, images and sequences of user actions.

How Do I Evaluate a Recommender Offline?

Measure recall at the candidate stage, and ranking metrics such as precision at 10 or NDCG on held-out interactions for the final list. Offline metrics are only a guide, because past logs reflect what the old system chose to show. Confirm important changes with an online A/B test.

Disclaimer: The content on this website reflects the views of contributing authors and not necessarily those of Generative AI Lab. This site may contain sponsored content, affiliate links, and material created with generative AI. Thank you for your support.

Generative AI Lab mascot
Generative AI Lab Editorial Team
Articles researched, written and edited by the Generative AI Lab team. About Generative AI Lab

Must read

- Advertisement -spot_img

More articles

- Advertisement -spot_img

Latest articles