Probability and Bayes' rule

A screening test for a disease is right about 90% of the time. Your result comes back positive. How worried should you be? A common answer is “about 90%”. For a rare condition the true answer can be closer to 9%. By the end of this lesson you will see exactly why, and you will have used the same reasoning to build a spam filter.

Probability as a share of cases

A probabilityProbabilityA number from 0 to 1 saying how likely something is. One way to read it: out of many similar cases, the share in which it happens.Open in glossary is a number from 0 to 1 that says how likely something is. One useful way to read it: imagine many cases and count. “The chance of rain is 30%” means that on days like this one, it rains on about 30 out of 100.

The interesting questions are usually narrower: how likely is A among the cases where B happened? That is a conditional probabilityConditional probabilityThe probability of A given that B happened, written P(A | B): the share of B cases that are also A. P(A | B) and P(B | A) are usually different numbers.Open in glossary, written $P(A \mid B)$ and read “the probability of A given B”. It is the share of B cases that are also A:

The order matters. $P(\text{positive} \mid \text{sick})$ asks: among sick people, what share test positive? $P(\text{sick} \mid \text{positive})$ asks: among people who test positive, what share are sick? The first describes the test. The second is what the patient wants to know. They can be wildly different.

A test for a rare condition

Three numbers describe the situation:

• The base rateBase rateHow common something is in the population before any evidence, such as the share of people who have a condition. Ignoring it makes rare events look far more likely after a positive test than they are.Open in glossary: how common the condition is. Here 1% of people have it.
• Sensitivity: the share of sick people the test catches, $P(\text{positive} \mid \text{sick})$. Here 90%.
• Specificity: the share of healthy people the test correctly clears, $P(\text{negative} \mid \text{healthy})$. Here 91%, so 9% of healthy people get a false alarm.

Now count. Of 1000 people, 10 are sick and 9 of them test positive. Of the 990 healthy people, 9% is about 89 who also test positive. Everyone with a positive result is in a group of 98, and only 9 of them are sick: about 9.2%.

prevalence = 0.01   # 1% of people have the condition
sensitivity = 0.90  # the test catches 90% of sick people
specificity = 0.91  # and clears 91% of healthy people

# Natural frequencies: the same numbers as counts out of 1000 people.
people = 1000
sick = people * prevalence
healthy = people - sick
true_pos = sick * sensitivity
false_pos = healthy * (1 - specificity)
print(f"{true_pos:.1f} sick and {false_pos:.1f} healthy people test positive")

# Bayes' rule: the share of positives who are really sick.
print(f"P(sick | positive) = {true_pos / (true_pos + false_pos):.3f}")

# Odds form: each positive multiplies the odds by the likelihood ratio
# (assuming repeat tests make independent mistakes).
odds = prevalence / (1 - prevalence)
ratio = sensitivity / (1 - specificity)
for n in range(1, 4):
    odds *= ratio
    print(f"positives in a row: {n}, P(sick) = {odds / (1 + odds):.3f}")

9.0 sick and 89.1 healthy people test positive
P(sick | positive) = 0.092
positives in a row: 1, P(sick) = 0.092
positives in a row: 2, P(sick) = 0.503
positives in a row: 3, P(sick) = 0.910

The false alarms win because there are so many more healthy people to make mistakes on. A small error rate applied to 990 people produces more positives than a large hit rate applied to 10. Ignoring this, and judging only by how accurate the test is, is called base rate neglect.

Counting whole people, as the grid does, makes this much easier to see than percentages. Psychologists Gerd Gigerenzer and Ulrich Hoffrage found that people solve problems like this far more often when the numbers are given as counts (“9 of 10 sick people test positive”) than as probabilities (Psychological Review, 1995).

You will meet $P(\text{sick} \mid \text{positive})$ again in Measuring a classifier, where it is called precision: of the cases flagged, how many were real.

Bayes’ rule

The counting you just did has a name. Bayes’ ruleBayes' ruleThe rule for turning P(evidence | hypothesis) into P(hypothesis | evidence): multiply by the prior and divide by the overall chance of the evidence.Open in glossary turns a probability in one direction into the other:

Here $H$ is a hypothesis (sick) and $E$ is the evidence (a positive test). Each piece has a name:

• $P(H)$ is the priorPriorWhat you believe about a hypothesis before seeing the evidence, as a probability. Bayes' rule combines it with the likelihood to give the posterior.Open in glossary: what you believed before the evidence, here the base rate.
• $P(E \mid H)$ is the likelihoodLikelihoodHow probable the observed evidence is if a hypothesis is true, P(evidence | hypothesis). Comparing it across hypotheses (the likelihood ratio) says how strongly the evidence favors one over another.Open in glossary: how expected the evidence is if the hypothesis is true, here the sensitivity.
• $P(H \mid E)$ is the posteriorPosteriorWhat you should believe about a hypothesis after seeing the evidence, P(hypothesis | evidence), computed from the prior and the likelihood by Bayes' rule.Open in glossary: what you should believe after seeing the evidence.
• $P(E)$ is the overall chance of the evidence, counting both ways it can happen: $0.9 \times 0.01 + 0.09 \times 0.99 \approx 0.098$.

Plugging in gives $0.9 \times 0.01 / 0.098 \approx 0.092$, the same 9.2% the counts gave. The rule is named after Thomas Bayes, an English minister whose essay on the problem was published in 1763, two years after his death, by his friend Richard Price.

Adding up many pieces of evidence: naive Bayes

A spam filter faces the same question with more evidence. Given the words in a message, how likely is it to be spam? Bayes’ rule says

where $\propto$ means “proportional to”: compute the same product for “not spam”, then scale the two so they add to 1. The hard part is $P(\text{words} \mid \text{spam})$: no training set contains every possible message. Naive BayesNaive BayesA classifier that applies Bayes' rule assuming the features are independent once the class is known, so each word (or feature) adds its own evidence. Simple and fast, but its probabilities tend to be overconfident.Open in glossary makes a bold simplification. It assumes that, once you know the class, each word appears independently of the others, so the probability of the message is a product of one probability per word:

Each $P(w \mid \text{spam})$ is easy to estimate: the share of all words in the spam training messages that are $w$. To avoid zeros for words a class has never seen, every count gets a small extra amount $\alpha$, called smoothingSmoothingAdding a small count to every possible word or value before turning counts into probabilities, so nothing unseen gets probability zero. Adding exactly 1 is called Laplace smoothing.Open in glossary:

Multiplying many small numbers is awkward, so the filter works with logarithms, which turn the product into a sum. Each word contributes its log ratio, $\ln\big(P(w \mid \text{spam}) / P(w \mid \text{not spam})\big)$, and the prior contributes $\ln\big(P(\text{spam}) / P(\text{not spam})\big)$. Positive totals point to spam, negative to not spam. That is exactly the odds form of Bayes’ rule, applied once per word.

import re
from collections import Counter

import numpy as np

spam = ["Win a free prize now", "Claim your free cash prize today",
        "Click here to win cash now", "You have won a free phone, click now"]
normal = ["Are we still on for lunch today?", "Can you send me the notes from class",
          "Running late, see you at the meeting", "Did we win the game? See you there",
          "Meeting moved to 3, free for lunch now?", "Happy birthday! Lunch is on me today"]

def words(text):
    return re.findall(r"[a-z0-9']+", text.lower())

counts = [Counter(w for m in normal for w in words(m)),
          Counter(w for m in spam for w in words(m))]
vocab = set(counts[0]) | set(counts[1])
alpha = 1.0  # smoothing: add 1 to every count

def p_spam(text, use_prior=True):
    # Start from the prior log-odds, then add one log ratio per word.
    log_odds = np.log(len(spam) / len(normal)) if use_prior else 0.0
    for w in words(text):
        if w not in vocab:
            continue  # never seen in training: skip it
        p1 = (counts[1][w] + alpha) / (sum(counts[1].values()) + alpha * len(vocab))
        p0 = (counts[0][w] + alpha) / (sum(counts[0].values()) + alpha * len(vocab))
        log_odds += np.log(p1 / p0)
    return 1 / (1 + np.exp(-log_odds))

for text in ["Free lunch today, see you there", "free now free now free now", "you win"]:
    print(f"{text!r}: {p_spam(text):.3f} (no prior: {p_spam(text, False):.3f})")

'Free lunch today, see you there': 0.082 (no prior: 0.118)
'free now free now free now': 0.995 (no prior: 0.997)
'you win': 0.457 (no prior: 0.558)

The default message, “Free lunch today, see you there”, contains free, a word with a strong spam lean. The filter still calls it 8.2% spam because lunch, see, you, and there all point the other way, and the prior adds a push of its own: 6 of the 10 training messages are not spam.

The independence assumption is clearly false: “click” and “here” travel together, and so do “see” and “you”. Naive Bayes classifies surprisingly well anyway, because it only needs the right class to come out on top, but its probabilities are often far too extreme. Repeating a word shows why: every copy is counted as new evidence. In 1998, Mehran Sahami and colleagues described a junk-mail filter built on a naive Bayesian classifier.