RugvAI Labs

Naive Bayes

How did Gmail know that the email subject "RE: URGENT — claim your prize now!!!" is spam, even though it had never seen that exact email before? Because of a 250-year-old equation from a Presbyterian minister named Thomas Bayes. The recipe: count how often each word appears in spam vs. real email, multiply the probabilities together, pick the bigger one. That's it. By the end of this lesson, you'll classify an email by hand using only multiplication and a small table — and you'll understand why a wildly unrealistic assumption ("every word is independent of every other word") gives a classifier that still beats elaborate deep models on small text datasets in 2026.

Learning Objectives

After this lesson, you will be able to:

Understand the 'naive' assumption — pretend features are independent *given the class* — and why this shortcut works surprisingly well even though it is technically wrong
Build the Naive Bayes formula step by step from Bayes' theorem and handle the 'never-seen-this-word-before' problem with Laplace smoothing
Know the three flavors of Naive Bayes (Gaussian, Multinomial, Bernoulli) and pick the right one for numbers vs. word counts vs. yes/no features
Explain why multiplying many small probabilities causes underflow and how log space solves it
Recognize when Naive Bayes outperforms more complex models — specifically in high-dimensional, low-data text classification scenarios

Prerequisites

Before You Start

Review

Why

Why This Matters

How Gmail catches 99.9% of spam -- Gmail's original spam filter was a Naive Bayes classifier that computed P(spam | words in email); despite its simplicity, it outperformed hand-written rules because it could learn from millions of user-reported spam examples and adapt to new spam tactics automatically
How your phone autocategorizes notifications -- when iOS or Android groups notifications into "social," "promotions," and "updates," it uses text classification at its core; Naive Bayes-style models are fast enough to classify every notification in real time without draining your battery
Why "naive" does not mean "bad" -- Naive Bayes assumes every feature is independent given the class label, which is almost never true in reality (the words "free" and "click" clearly co-occur in spam); yet empirically, this incorrect assumption often leads to surprisingly accurate classifiers because the ranking of probabilities matters more than their exact values
Build this --> Build a spam classifier that processes actual message text, counts word frequencies, applies Bayes' theorem with Laplace smoothing, and classifies messages as spam or ham -- test it on messages from your own inbox

Don't worry if the math looks intense — Naive Bayes is really just multiplying a bunch of simple probabilities together and picking the biggest answer. Once you see it work on a spam filter example, it will click!

#From Bayes' Theorem to a Classifier

Intuition

Build Your Intuition

You are a detective. You need to decide if a mysterious email is spam or not. You look at the evidence: the words in the email. Each word is a clue.

"FREE" -- that is suspicious (60% of spam contains "free," but only 5% of legitimate emails). "WINNER" -- very suspicious. "MEETING" -- that is normal (rare in spam, common in real emails). "CLICK" -- suspicious again.

Bayes' theorem tells you how to combine all these clues into a single probability: P(spam | all the words). The problem is that computing the exact probability requires knowing how all words co-occur together, which is impossibly complex with thousands of words.

The "naive" assumption cuts through this complexity: once you have fixed the class, pretend the words are independent clues. Formally that is conditional independence — P(x₁, x₂, ..., xₙ | spam) = P(x₁ | spam) × P(x₂ | spam) × ... — not the much stronger claim that the words are independent outright. The distinction matters: "free" and "click" are obviously correlated in general, and Naive Bayes never claims otherwise. It only assumes that within the spam bucket, knowing "free" appeared tells you nothing extra about whether "click" appeared. That is still wrong, but it is a far more modest kind of wrong — and it works remarkably well because you only need the ranking to be correct (spam probability > ham probability), not the exact numbers.

We want to classify a document d (with features x1, x2, ..., xn) into one of several classes (like spam or ham). By Bayes' theorem:

P(c|x_1, x_2, \ldots, x_n) = \frac{P(x_1, x_2, \ldots, x_n|c) \cdot P(c)}{P(x_1, x_2, \ldots, x_n)}

The denominator P(x1, ..., xn) is the same for all classes, so when comparing classes we only need the numerator. The classification rule becomes: pick the class with the highest numerator.

#The Naive Assumption

Computing P(x1, x2, ..., xn | c) exactly requires knowing the joint distribution of all features given the class -- exponentially many parameters. With 10,000 vocabulary words, you would need to estimate 2^10000 probabilities. Impossible.

The naive assumption: all features are conditionally independent given the class.

P(x_1, x_2, \ldots, x_n | c) \approx \prod_{i=1}^{n} P(x_i | c)

So the full Naive Bayes classifier is:

Naive Bayes Spam Filter — Toggle Words and Watch P(spam) MultiplyInteractive

Loading visualization...

\hat{c} = \arg\max_c \, P(c) \prod_{i=1}^{n} P(x_i | c)

Try it! Open the Python REPL and type these lines yourself. Build a one-line spam classifier: from sklearn.naive_bayes import MultinomialNB; from sklearn.feature_extraction.text import CountVectorizer; v = CountVectorizer(); X = v.fit_transform(["free money now","meeting tomorrow","win prize","project update"]); y = [1,0,1,0]; m = MultinomialNB().fit(X,y); print(m.predict(v.transform(["free prize"]))) — it predicts spam!

#Quick Spam Classifier Demo

pythonplayground.py · Pyodide

Tests · Verify 'free money click now' is classified as SPAM. Verify 'project meeting tomorrow' is classified as HAM.

#Spam Filter Walkthrough

Let us build a spam classifier from scratch using real word frequencies.

Training data summary (from 1,000 emails: 400 spam, 600 ham):

Word	P(word \| spam)	P(word \| ham)
free	0.60	0.05
winner	0.40	0.01
click	0.50	0.08
meeting	0.02	0.30
now	0.45	0.15

Priors: P(spam) = 400/1000 = 0.40, P(ham) = 600/1000 = 0.60.

What Do You Think?

An email contains 'FREE WINNER CLICK NOW'. Spam or not?

#Step 1: Compute the Spam Score

P(spam) x P(free|spam) x P(winner|spam) x P(click|spam) x P(now|spam)

= 0.40 x 0.60 x 0.40 x 0.50 x 0.45

= 0.40 x 0.054 = 0.0216

#Step 2: Compute the Ham Score

P(ham) x P(free|ham) x P(winner|ham) x P(click|ham) x P(now|ham)

= 0.60 x 0.05 x 0.01 x 0.08 x 0.15

= 0.60 x 0.000006 = 0.0000036

#Step 3: Classify

Spam score: 0.0216 Ham score: 0.0000036

Spam score is ~6,000x larger. This email is overwhelmingly classified as spam.

To get actual probabilities: P(spam|words) = 0.0216 / (0.0216 + 0.0000036) = 99.98%

Every word in this email is strong evidence for spam. The combined effect is devastating.

#Laplace Smoothing: The Zero-Frequency Problem

What happens if the word "cryptocurrency" appears in a test email but was never seen in the training spam? P(cryptocurrency | spam) = 0, and since we multiply all probabilities, the entire product becomes zero -- regardless of all other evidence. One unseen word kills the classifier.

Laplace smoothing (add-one smoothing) fixes this by adding a small count to every word:

P(x_i | c) = \frac{\text{count}(x_i, c) + \alpha}{\text{count}(c) + \alpha \cdot |V|}

With alpha = 1 (add-one smoothing), a word that never appeared in spam gets a probability of 1/(total_spam_words + vocab_size) instead of 0. This is a crucial practical detail -- without smoothing, Naive Bayes breaks on any email containing new words.

#Three Flavors of Naive Bayes

#Multinomial Naive Bayes

Best for text classification where features are word counts or frequencies. This is what we used in the spam example. It models how many times each word appears.

Use for: spam filtering, sentiment analysis, document categorization, news topic classification.

#Bernoulli Naive Bayes

Features are binary (word present or absent, not counts). It also considers the absence of words as evidence. If "meeting" is absent, that is evidence against ham.

Use for: short text classification, feature presence/absence, binary survey data.

#Gaussian Naive Bayes

Features are continuous and assumed normally distributed. For each class, it estimates the mean and variance of each feature.

Use for: iris flower classification, medical diagnosis with continuous lab values, any continuous feature set.

Now let's actually run Multinomial Naive Bayes on text. The cell below tokenizes a small set of restaurant reviews, fits the model, and prints the per-word log-probabilities so you can see exactly which words pull toward "positive" versus "negative".

Loading visualization...

Quick check

Naive Bayes assumes features are CONDITIONALLY INDEPENDENT given the class. For a spam classifier, what does that assumption claim about the words 'free' and 'click'?

#Interactive: Watch Naive Bayes Classify Data

Try it: Watch Naive Bayes classify data in real timeInteractive

Loading visualization...

Try this: Generate two clusters of data and watch Naive Bayes draw its decision boundary. Notice how the boundary is always a straight line (for Gaussian NB with equal variance) or a curve. Compare it to logistic regression -- the boundaries look similar because both are linear classifiers in feature space. Then try making the clusters overlap and see how the prior affects where the boundary falls.

Interactive Lab

Switch to the Naive Bayes algorithm to see its boundary on the same dataset. Compare with logistic regression — both linear, but driven by very different math under the hood.

#Why Does Naive Bayes Actually Work?

The independence assumption is clearly wrong. "New" and "York" co-occur. "Free" and "click" co-occur in spam. So why does it work?

#The Deeper Reason "Naive" Works

Despite assuming word independence (clearly wrong -- "New York" is not independent), Naive Bayes works surprisingly well because it only needs the RANKING of probabilities to be correct, not the exact values. Consider: if the true P(spam|email) is 0.92 and Naive Bayes estimates 0.73, the classification is still correct -- spam wins. The probabilities are wrong, but the ordering is right.

This is analogous to a judge ranking contestants. Even if the judge's scores are miscalibrated (giving everyone lower scores than they deserve), as long as the ranking is preserved, the right contestant wins. Naive Bayes is a consistently miscalibrated judge who still usually picks the right winner.

#Try It Yourself

pythonplayground.py · Pyodide

# Build a Naive Bayes spam classifier from scratch
import numpy as np

# Training data: word frequencies in spam vs ham
training_data = {
  "spam": {
      "count": 400,
      "words": {"free": 240, "winner": 160, "click": 200, "now": 180,
                "meeting": 8, "report": 12, "hello": 40, "urgent": 140,
                "prize": 120, "offer": 160, "limited": 100, "deal": 90},
      "total_words": 1450,
  },
  "ham": {
      "count": 600,
      "words": {"free": 30, "winner": 6, "click": 48, "now": 90,
                "meeting": 180, "report": 150, "hello": 240, "urgent": 30,
                "prize": 6, "offer": 30, "limited": 18, "deal": 24},
      "total_words": 852,
  },
}

vocab_size = 12  # unique words
alpha = 1  # Laplace smoothing

def get_word_prob(word, class_name):
  data = training_data[class_name]
  word_count = data["words"].get(word, 0)
  return (word_count + alpha) / (data["total_words"] + alpha * vocab_size)

def classify(email_words):
  total_emails = training_data["spam"]["count"] + training_data["ham"]["count"]

# Use log probabilities to avoid underflow
  log_spam = np.log(training_data["spam"]["count"] / total_emails)
  log_ham = np.log(training_data["ham"]["count"] / total_emails)

for word in email_words:
      log_spam += np.log(get_word_prob(word, "spam"))
      log_ham += np.log(get_word_prob(word, "ham"))

# Convert to probability using log-sum-exp trick
  max_log = max(log_spam, log_ham)
  p_spam = np.exp(log_spam - max_log)
  p_ham = np.exp(log_ham - max_log)
  total = p_spam + p_ham

return {
      "prediction": "SPAM" if log_spam > log_ham else "HAM",
      "spam_prob": f"{p_spam / total * 100:.2f}%",
      "ham_prob": f"{p_ham / total * 100:.2f}%",
  }

# Test emails
tests = [
  ["free", "winner", "click", "now"],
  ["meeting", "report", "hello"],
  ["free", "meeting", "report"],
  ["urgent", "prize", "limited", "offer"],
  ["hello", "free", "deal"],
]

for email in tests:
  result = classify(email)
  print(f'"{" ".join(email)}" => {result["prediction"]}')
  print(f'  P(spam)={result["spam_prob"]}, P(ham)={result["ham_prob"]}')
  print()

# CHALLENGE: Add the word "unsubscribe" to the vocabulary
# with spam frequency 100 and ham frequency 10.
# How does it change the classification of "free meeting report"?

Tests · Verify 'free winner click now' is classified as spam. Verify 'meeting report hello' is classified as ham. Check that probabilities sum to approximately 100%.

On the Optimality of the Simple Bayesian Classifier under Zero-One Loss

Pedro Domingos and Michael Pazzani (1997)

The seminal paper explaining why Naive Bayes works despite the clearly wrong independence assumption. Shows that the classifier only needs to get the ranking right, not the exact probabilities, and that the independence assumption often does not affect the ranking.

#Looking Ahead: Discriminant Analysis

Naive Bayes is one member of a larger family — generative classifiers — all of which model P(X | y) · P(y) and apply Bayes' rule to recover P(y | X). They only differ in what they assume about P(X | y). Gaussian Naive Bayes assumes per-class Gaussians with diagonal covariance (features conditionally independent). Drop that diagonal restriction and you get Linear Discriminant Analysis (LDA) — same shared covariance across classes, but full matrices. Let each class wear its own covariance and you get Quadratic Discriminant Analysis (QDA). These are the next lesson — a discriminative-looking cousin family of Gaussian Naive Bayes, with linear vs. quadratic boundaries falling out of one assumption. See Linear & Quadratic Discriminant Analysis for the full story, including Fisher's 1936 criterion for supervised dimensionality reduction.

#When to Use Naive Bayes vs. Alternatives

Situation	Reach for	Why
Text classification with word counts (spam, sentiment, topic)	Multinomial Naive Bayes	Default workhorse; trains in milliseconds on millions of docs
Binary feature vectors (presence/absence)	Bernoulli Naive Bayes	Treats "absence" as informative evidence
Continuous features, classes roughly Gaussian	Gaussian NB (or LDA — see next lesson)	NB's diagonal-Σ assumption is cheap; LDA captures within-class correlations
Continuous features, many correlated, > 100 features	Logistic regression + L2	NB double-counts correlated evidence and gets overconfident
You need probability calibration for downstream decisions	Logistic regression (or calibrate NB with Platt scaling)	NB probabilities are notoriously over-confident
Tabular data with mixed types and rich interactions

#Key Takeaways

Naive Bayes is Bayes' theorem with a strong independence shortcut -- it classifies by computing P(class) times the product of P(feature|class) for each feature, picking the class with the highest score
The 'naive' assumption is wrong but works -- features are rarely independent, but the classifier only needs correct ranking (which class scores highest), not exact probabilities; this makes it a surprisingly strong baseline
Laplace smoothing prevents zero probabilities -- adding a small count (alpha) to every feature ensures that one unseen word does not zero out the entire probability; without it, Naive Bayes breaks on new vocabulary
Three variants for different data types -- Multinomial NB for word counts (text), Bernoulli NB for binary features, Gaussian NB for continuous features; Multinomial NB is the workhorse of text classification
Log probabilities prevent underflow -- always work in log space when implementing Naive Bayes to avoid multiplying many tiny numbers into zero

#Quick Check

Quick Check1 / 5

Why is it called 'Naive' Bayes?

Next up: Linear & Quadratic Discriminant Analysis -- Gaussian Naive Bayes' more flexible cousin, where dropping the diagonal-covariance assumption gives you linear (LDA) and quadratic (QDA) decision boundaries, and Fisher's 1936 trick repurposes the same math for supervised dimensionality reduction.

Your Reflection

Continue Learning

Discussion

Naive Bayes

After this lesson, you will be able to:

Before You Start

#From Bayes' Theorem to a Classifier

#The Naive Assumption

#Quick Spam Classifier Demo

#Spam Filter Walkthrough

#Step 1: Compute the Spam Score

#Step 2: Compute the Ham Score

#Step 3: Classify

#Laplace Smoothing: The Zero-Frequency Problem

#Three Flavors of Naive Bayes

#Multinomial Naive Bayes

#Bernoulli Naive Bayes

#Gaussian Naive Bayes

#Interactive: Watch Naive Bayes Classify Data

#Why Does Naive Bayes Actually Work?

Deep Dive

#The Deeper Reason "Naive" Works

#Try It Yourself

#Looking Ahead: Discriminant Analysis

#When to Use Naive Bayes vs. Alternatives

#Key Takeaways

#Quick Check

Deep Dive

Related Lessons