Choosing a threshold

A simulated spam filter gives each of 1000 emails a score from 0 to 1. Emails scoring at or above the threshold are flagged as spam.

Spam (above the line)
Good email (below)
Threshold (drag it)

Confusion matrix
	Flagged	Delivered
Really spam	275spam caught (true positives)	25spam missed (false negatives)
Really good	70good mail flagged (false positives)	630good mail delivered (true negatives)

ROC curve. The dot is the current threshold; the dashed diagonal is random guessing.

Threshold0.50

Model quality (how far apart the scores are)2.5

Share of emails that are spam30%

Accuracy90.5%

Precision79.7%

Recall91.7%

False positive rate10.0%

Area under ROC0.968

Challenges

0 of 3 complete

Make the accuracy look excellent, above 97%, while the filter catches less than a fifth of the spam. (not yet)
Get precision and recall both to 90% or more. (not yet)
Keep the threshold at 0.5 and find a mix of email where the filter still catches over 80% of spam, yet fewer than half of its flags are really spam. (not yet)

For comparison, a filter that flags nothing at all would be 70.0% accurate on this mix of email, and catch no spam.

Try this

Sweep the threshold from left to right and watch precision and recall trade places.
Lower the share of spam to 2% and compare accuracy with the filter that flags nothing.
Set model quality to 0, then to 5, and watch the ROC curve.

Precision, recall, and AUC in NumPyPython with NumPy

import numpy as np

# A spam filter's scores for 12 emails, and the truth: 1 = spam, 0 = good.
scores = np.array([0.95, 0.85, 0.80, 0.70, 0.55, 0.40,
                   0.60, 0.45, 0.35, 0.20, 0.10, 0.05])
labels = np.array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0])

for threshold in (0.3, 0.5, 0.75):
    flagged = scores >= threshold          # at or above the threshold: spam
    tp = np.sum(flagged & (labels == 1))   # spam caught
    fp = np.sum(flagged & (labels == 0))   # good mail flagged
    fn = np.sum(~flagged & (labels == 1))  # spam missed
    tn = np.sum(~flagged & (labels == 0))  # good mail delivered
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    accuracy = (tp + tn) / len(labels)
    print(f"threshold {threshold}: TP {tp} FP {fp} FN {fn} TN {tn}, "
          f"precision {precision:.2f}, recall {recall:.2f}, accuracy {accuracy:.2f}")

# Area under the ROC curve: the chance that a random spam email
# scores higher than a random good one (ties count half).
spam, good = scores[labels == 1], scores[labels == 0]
pairs = spam[:, None] - good[None, :]
auc = np.mean(pairs > 0) + 0.5 * np.mean(pairs == 0)
print(f"AUC {auc:.3f}")

threshold 0.3: TP 6 FP 3 FN 0 TN 3, precision 0.67, recall 1.00, accuracy 0.75
threshold 0.5: TP 5 FP 1 FN 1 TN 5, precision 0.83, recall 0.83, accuracy 0.83
threshold 0.75: TP 3 FP 0 FN 3 TN 6, precision 1.00, recall 0.50, accuracy 0.75
AUC 0.917