Part 2 of 4 · Chapter 4 of 4

Measuring a Model

Accuracy is the first number everyone reaches for and the first one that will mislead you. Drag a decision threshold and watch precision and recall trade against each other.

Beginner13 min read

Accuracy is the first number everyone reaches for, and it is the first number that will mislead you. This lesson is about the two measurements that replace it, why they pull against each other, and how to decide which one your problem actually cares about.

A model that does nothing, and scores 99.7%

Suppose you are building a fraud detector. In your data, 3 transactions in every 1,000 are fraudulent. You train a model, and it comes back with 99.7% accuracy. That sounds like a finished project.

Now here is a model that also scores 99.7%, which you can write in one line:

def predict(transaction):

return "not fraud"

It is right 997 times out of 1,000, because it never has to do anything. It also catches exactly zero fraud, which was the entire point of building it.

The trap: accuracy counts every prediction equally. When one class is rare, the model can score brilliantly on the common case and be useless on the case you care about. You need numbers that look only at the rare thing.

Four ways a yes-or-no prediction can land

Before any formula, get the outcomes straight. Your model says yes or no. Reality is yes or no. That is four combinations, and each one costs something different.

True positive

Flagged it, and it was really fraud. The transaction gets stopped.

False negative

Said it was fine, and it was fraud. The money is gone and nobody knows yet.

False positive

Flagged it, and it was legitimate. A real customer gets their card declined.

True negative

Said it was fine, and it was. This is almost everything, and it is why accuracy lies.

Arranged in a grid, these four counts are called a confusion matrix. It is not a thing to memorise — it is just these four numbers in a box. Precision and recall are each built from three of them.

Precision: of everything I flagged, how much was real?

Precision looks only at the cases your model raised its hand for, and asks what fraction of them deserved it.

precision = TP / (TP + FP)

Its failure mode is crying wolf. Low precision means most of your alerts are noise, and the humans downstream will start ignoring all of them — which quietly destroys the value of the alerts that were correct. Precision is owned entirely by false positives.

Recall: of everything real, how much did I catch?

Recall ignores your alerts and looks at reality instead: of all the fraud that actually happened, how much did the model find?

recall = TP / (TP + FN)

Its failure mode is missing the thing. Low recall means the problem is still happening and your dashboard is calm about it. Recall is owned entirely by false negatives — and false negatives are invisible unless you go looking for them.

Notice that the one-line fraud model from earlier has a recall of 0. That is the number that would have caught the problem immediately.

The dial between them

Here is the part that makes this interesting: you can max out either metric on its own, trivially, and the result is useless both times. Flag every single case and recall is a perfect 1.0. Flag only the one case you are most certain about and precision is a perfect 1.0. Neither model is worth shipping.

What connects them is the threshold. A classifier does not really output yes or no — it outputs a confidence score, and you choose the cutoff above which that score becomes a yes. Move the cutoff and you are trading one metric for the other.

Drag the threshold

Every case the model scored, sorted left to right by how confident it was. Real positives sit above the line, real negatives below it. Everything to the right of the threshold is what the model flags.

The four outcomes

131

flagged, and real

missed a real one

false alarm

277

correctly ignored

160 real positives and 340 real negatives in total.

Precision

0.68

Of everything I flagged, how much was real?

131 / (131 TP + 63 FP) = 0.68

Recall

0.82

Of everything real, how much did I catch?

131 / (131 TP + 29 FN) = 0.82

F1

0.74

The harmonic mean of the two. It only goes up when both go up.

Two things worth noticing

Drag the threshold slowly from left to right. Recall falls smoothly and predictably — every step right abandons a few more real positives, and they never come back.

Precision does not behave nearly as politely. It climbs, but it jumps around as it goes, especially at the high end. That is not a bug in the chart. Once you are flagging only a handful of cases, a single mistake swings the ratio hard. It is exactly why “just pick a high-precision threshold” is bad advice: up there, the number is measured on so few cases that it barely means anything.

Why F1 is a harmonic mean

F1 combines the two into one number so you can compare models at a glance. It deliberately is not a plain average. Consider the flag-everything model: precision 0.32, recall 1.0. Averaged normally, that is a respectable-looking 0.66 for a model with no judgement at all.

F1 = 2 · (precision · recall) / (precision + recall)

The harmonic mean is dragged toward the smaller of the two numbers, so it punishes lopsidedness. Take the extreme case — precision 1.0, recall 0.0. A normal average reports 0.5. F1 reports 0.

So which one do you optimise?

There is no general answer, and anyone who gives you one is skipping the actual question: which mistake is more expensive for the people using this?

Screening for cancer

Optimise Recall

A false alarm costs one more test and a frightening afternoon. A miss costs a life. You accept a pile of false positives to make sure almost nothing real slips through.

Filtering spam

Optimise Precision

A spam email that reaches the inbox is a mild annoyance. A job offer routed to the spam folder is a disaster the user may never discover. Only flag what you are sure about.

Ranking search results

Optimise Precision, but only at the top

Nobody reads result 400, so recall over the whole index is close to meaningless. What matters is whether the first handful are right — measured as precision@k.

Same maths, three different answers. Choosing a metric is a product decision that happens to be expressed in numbers — it is not something the maths can decide for you.

Computing these yourself

You will rarely write these by hand. In Python:

from sklearn.metrics import precision_score, recall_score

from sklearn.metrics import f1_score, confusion_matrix

precision = precision_score(y_true, y_pred)

recall = recall_score(y_true, y_pred)

f1 = f1_score(y_true, y_pred)

tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()

The thing worth building a habit around is not the function call. It is printing the confusion matrix before you trust any single score.

Key takeaways

Accuracy hides failure whenever one class is rare. Check recall before you celebrate.
Precision is about your alerts. Recall is about reality.
The threshold is the dial between them, and moving it always costs one to buy the other.
F1 is harmonic so that being terrible at one metric cannot be averaged away.
Which metric matters is decided by which mistake is more expensive — not by the maths.