Evaluation Metrics

I avoided evaluation metrics for an embarrassingly long time. Every project, I'd train a model, call accuracy_score, see a comforting number like 98.7%, and pat myself on the back. Then one day I deployed a fraud detector that scored 99% accuracy in testing. In production, it caught zero fraud. Zero. It had learned to predict "not fraud" for every single transaction, and because only 1% of transactions were actually fraudulent, that was enough to look brilliant on paper. That was the day the number I trusted betrayed me, and I finally sat down to understand what it was actually measuring. Here is that dive.

Evaluation metrics are the scorecards we use to judge how well a model performs. They date back to signal detection theory in the 1940s (radar operators trying to distinguish enemy planes from flocks of birds), and they've been refined across medicine, information retrieval, and machine learning ever since. The core question hasn't changed: how do we measure whether our predictions are any good?

Before we start, a heads-up. We're going to be building up from a two-by-two table all the way to calibration curves, ranking metrics, and the hard-earned wisdom of what breaks in production. We'll talk about some formulas, but you don't need any of them memorized in advance. We'll construct each one from scratch, and by the time we write it down, it'll feel obvious. We'll add what we need, one piece at a time.

This isn't a short journey, but I hope you'll be glad you came.

The Four Buckets

Every classification metric you'll encounter in your career is some arithmetic on four numbers. Every one. Before we get to any metric, we need to see where these numbers come from, because once you can see the four buckets, the rest of evaluation metrics is an exercise in deciding which ratios matter to you.

Here's the scenario we'll carry through this entire section. You work at a bank. Your job is to build a fraud detection system. Out of every 10,000 transactions that flow through the system each day, about 100 are fraudulent — a 1% fraud rate. You've trained a model. Now you run it against your test set and it makes a prediction on each of the 10,000 transactions: "fraud" or "not fraud." Some of those predictions will be right, some will be wrong. That gives us four possible outcomes.

That's the confusion matrix. The name is apt — most of the confusion in evaluation metrics comes from not knowing which of these four cells a metric cares about and which it ignores.

Let's put concrete numbers to our scenario. Say the model catches 85 of the 100 frauds, misses 15, and also wrongly flags 50 legitimate transactions as fraud.

import numpy as np
from sklearn.metrics import confusion_matrix

y_true = np.array([1]*100 + [0]*9900)

y_pred = np.zeros(10000, dtype=int)
y_pred[:85] = 1      # catches 85 of 100 frauds
y_pred[100:150] = 1  # 50 false alarms on legit transactions

cm = confusion_matrix(y_true, y_pred)
# [[9850,   50],    ← TN=9850, FP=50
#  [  15,   85]]    ← FN=15,   TP=85

Four cells. That's it. Everything that follows is a question about which cells to put in the numerator and which in the denominator.

The Accuracy Trap

The most natural thing to compute is accuracy: how many did we get right out of everything?

Accuracy = (TP + TN) / (TP + TN + FP + FN)

In our fraud scenario: (85 + 9850) / 10000 = 99.35%. That sounds fantastic. And here's where the trap snaps shut. A model that does absolutely nothing — one that prints return "not fraud" for every single transaction, a model that literally has no idea what fraud looks like — scores 99.0%. Our "good" model's improvement over total ignorance is 0.35 percentage points.

I'll be honest — the first time I actually worked through this arithmetic, I felt like I'd been lied to for years. All those accuracy numbers I'd been reporting in meetings suddenly felt hollow. The problem is structural: accuracy treats every correct prediction as equally important. Getting a legitimate transaction right and catching a fraud are both worth one point. But in a world where 99% of transactions are legitimate, being right about the majority class drowns out everything that matters.

Think of it like a courtroom where the judge always says "not guilty." In a world where 99% of defendants are innocent, that judge has 99% accuracy. That judge is useless. Accuracy works when the classes are roughly balanced — cats versus dogs at 50/50. The moment one class dominates, accuracy becomes a metric that rewards laziness. We need something better.

Precision — The Courtroom Standard

Let's stay in the courtroom for a moment. One philosophy of justice says: "It is better that ten guilty persons escape than that one innocent suffer." That philosophy is about precision.

Precision = TP / (TP + FP)

Of all the transactions our model flagged as fraud, how many actually were? In our scenario: 85 / (85 + 50) = 63%. About 4 in every 10 alarms are false. If you have a human fraud analyst who has to investigate every flag, a precision of 63% means they spend a third of their day chasing ghosts. That's expensive, and it erodes trust in the system. After a few weeks of false alarms, the analyst starts ignoring flags altogether.

Precision matters most when the cost of a false alarm is high. A spam filter that sends a legal contract to junk — that's a false positive, and it could cost a deal. A content moderation system that takes down a legitimate post — that's a false positive, and it erodes user trust. A criminal justice system that convicts an innocent person — that's a false positive, and the cost is a human life.

The limitation of precision is that it's easy to game. A model can achieve 100% precision by being absurdly cautious — only flag something as fraud when you're 99.99% certain. You'll flag three transactions a year, get all three right, and miss the other 97 frauds. Precision alone is half the story.

Recall — The Cancer Screening Standard

The opposite philosophy says: "Never let a guilty person walk free, no matter the cost." That's recall.

Recall = TP / (TP + FN)

Of all actual frauds, how many did we catch? Our model: 85 / (85 + 15) = 85%. Fifteen fraudulent transactions slipped through. At, say, $5,000 per fraud, that's $75,000 in losses the model failed to prevent today alone.

Recall matters most when missing a positive is dangerous or expensive. Cancer screening — a false negative is a tumor that goes undetected, and the patient doesn't get treatment until it's too late. Airport security — a false negative is a threat that gets through. Manufacturing quality control — a false negative is a defective part that ships to a customer.

And recall has the opposite gaming problem. You can get 100% recall trivially: flag every single transaction as fraud. You'll catch all 100 frauds. You'll also flag 9,900 legitimate transactions, and your precision will be 100/10000 = 1%. Your fraud team quits. Your customers cancel their cards. Recall alone is the other half of the story.

The Smoke Detector Dial

Here's the thing that ties precision and recall together mechanically, and it took me a while to feel it in my bones rather than read it off a slide.

Most classifiers don't actually output a hard yes or no. They output a probability — a score between 0 and 1 that represents how suspicious a transaction looks. A score of 0.92 means the model is fairly convinced it's fraud. A score of 0.15 means it's fairly convinced it's legitimate. To get an actual prediction, you pick a threshold and say "anything above this threshold, I'll call fraud."

Think of a smoke detector with a sensitivity dial. Crank the dial all the way up: it catches every fire, every smoldering wire, every hint of smoke. It also goes off when you make toast, when you take a hot shower, when you boil water. That's high recall, low precision. Now dial it all the way down: it never goes off for toast or showers. It also doesn't go off until the house is fully engulfed. That's high precision, low recall.

There is no setting that gives you both. This isn't a flaw in the metric — it's a fundamental property of any system that has to make a binary decision from a continuous score. The right setting depends entirely on which mistake is more dangerous in your specific situation. And that's a business question, not a statistics question.

We'll come back to this smoke detector and its dial when we talk about threshold selection. For now, the question is: can we somehow compress the precision–recall tradeoff into a single number?

F1 and the Harmonic Mean Trick

The F1 score tries to give you one number that captures both precision and recall. It uses the harmonic mean:

F1 = 2 × (precision × recall) / (precision + recall)

Why harmonic mean instead of a plain average? This is worth pausing on, because it's one of those "why" questions that separates someone who memorized the formula from someone who understands what it's doing.

Suppose a model has precision of 100% and recall of 1%. The arithmetic mean (the regular average) would be (1.0 + 0.01) / 2 = 50.5%. That sounds pretty okay, right? Half decent? But that model catches 1% of all positives — it's essentially broken. The harmonic mean gives 2 × (1.0 × 0.01) / (1.0 + 0.01) = 0.0198, or roughly 2%. That number correctly screams that something is deeply wrong. The harmonic mean is bounded by the smaller value — it refuses to let a strong number on one side paper over a disaster on the other.

Our fraud model: F1 = 2 × (0.63 × 0.85) / (0.63 + 0.85) = 0.72. Decent, not great. The 63% precision is dragging it down.

Now, F1 treats precision and recall as equally important. In many real situations, they aren't. A cancer screening test should lean heavily toward recall — missing a tumor is worse than sending someone for an extra biopsy. A spam filter should lean toward precision — filtering a real email into junk is worse than letting some spam through. For these cases, there's F-beta:

F_β = (1 + β²) × (P × R) / (β² × P + R)

The β parameter controls the leaning. β = 2 makes recall twice as important as precision — good for fraud detection and medical screening. β = 0.5 makes precision twice as important — good for spam filters and content moderation. F1 is F-beta with β = 1. The way to think about β² is that it's the ratio of how much you care about recall relative to precision.

The limitation of F1 (and all the F-beta family) is that it completely ignores true negatives. It doesn't care how well you classify the majority class. For most imbalanced problems, that's fine — you already know the model can identify legitimate transactions. But it does mean F1 can sometimes tell an incomplete story, and that gap is what motivates the next metric we'll meet after our rest stop.

Rest Stop

Congratulations on making it this far. You can stop here if you want.

You now have a solid mental model of evaluation: a confusion matrix with four cells, accuracy and its trap, precision for when false alarms are expensive, recall for when misses are dangerous, the tradeoff between them (the smoke detector dial), and F1 as a single-number summary. That's genuinely useful. You can walk into most data science conversations and hold your own with what you know right now.

It doesn't tell the complete story, though. We haven't talked about what happens when you need to compare models across all possible thresholds (ROC and PR curves). We haven't talked about what happens when your model's confidence scores are lies (calibration). We haven't talked about regression, ranking, or the sneaky way metrics break in production. And we haven't talked about the one metric that's hardest to game — MCC.

The short version, if you want to bail: use F1 for balanced-cost binary classification, PR-AUC for imbalanced data, RMSE for regression where big misses matter, and always connect the metric back to what it costs when the model is wrong. There. You're 80% of the way there.

But if the discomfort of not knowing what's underneath is nagging at you, read on.

Threshold Selection — The Dial You Actually Turn

Let's go back to our smoke detector. We said most classifiers output a continuous score, and you pick a threshold to turn that score into a decision. The default threshold in most libraries is 0.5. I need to be blunt about this: 0.5 is arbitrary, and in my experience it's almost never optimal.

In our fraud detector, the model assigns every transaction a suspicion score between 0 and 1. At a threshold of 0.5, we flag anything above 50% suspicion. But think about the asymmetry: a missed fraud costs $5,000 in chargebacks, while a false alarm costs maybe $50 in analyst time. If misses cost 100× more than false alarms, why are we splitting the threshold down the middle?

Drop the threshold to 0.2 — flag anything with a 20%+ suspicion score. You'll catch more fraud (recall goes up), you'll also trigger more false alarms (precision goes down), but the total dollar cost might drop dramatically because each caught fraud saves $5,000 while each extra false alarm only costs $50. The right threshold is where your total business cost is minimized. If you can put dollar amounts on false positives and false negatives, you can compute this directly: total_cost = cost_FN × FN + cost_FP × FP. Sweep through thresholds, pick the one where total_cost is smallest. That's the dial setting your smoke detector actually needs.

from sklearn.metrics import precision_recall_curve
import numpy as np

# Find the threshold that gives at least 90% recall
precisions, recalls, thresholds = precision_recall_curve(y_true, y_scores)
idx = np.where(recalls >= 0.90)[0][-1]
print(f"For 90% recall: threshold = {thresholds[idx]:.3f}, "
      f"precision = {precisions[idx]:.3f}")

This is one of those things that seems obvious once someone says it out loud, but I've seen production systems running on the default 0.5 for years because no one thought to ask "what does a mistake actually cost?"

ROC-AUC — Ranking Quality Across All Thresholds

Threshold selection gives you the best setting for a specific cost structure. But what if you want to compare two models before you've decided on a threshold? What if you want to say "model A is better than model B, regardless of where we set the dial"? That's what the ROC curve is for.

The ROC curve (Receiver Operating Characteristic — the name is a relic from WWII radar, which is kind of wonderful) plots True Positive Rate (that's recall) on the y-axis against False Positive Rate on the x-axis, at every possible threshold. A perfect model hugs the top-left corner — perfect recall, zero false positives, at every threshold. A model that's flipping coins traces the diagonal line from bottom-left to top-right.

AUC (Area Under the ROC Curve) collapses that entire curve into one number between 0 and 1. The intuitive interpretation is wonderfully concrete: pick a random fraud and a random legitimate transaction. AUC is the probability that the model assigns a higher suspicion score to the fraud. AUC = 0.5 means the model can't tell them apart — coin flip. AUC = 0.85 means 85% of the time, a random fraud will look more suspicious than a random legitimate transaction.

from sklearn.metrics import roc_auc_score

np.random.seed(42)
y_true = np.array([1]*100 + [0]*9900)
scores_fraud = np.random.beta(5, 2, size=100)
scores_legit = np.random.beta(1.5, 5, size=9900)
y_scores = np.concatenate([scores_fraud, scores_legit])

auc = roc_auc_score(y_true, y_scores)
print(f"AUC: {auc:.3f}")  # ~ 0.96

ROC-AUC is the standard for comparing models when classes are reasonably balanced. But here's the catch, and this one bit me in production more than once: on heavily imbalanced data, ROC-AUC can be dangerously optimistic.

The False Positive Rate is FP / (FP + TN). When you have 99,000 legitimate transactions, even 500 false positives gives an FPR of only 0.5%. The curve looks beautiful. But 500 false alarms will absolutely bury your operations team. ROC-AUC is blind to this because it's divided by that massive denominator of true negatives. The curve smiles at you while your team drowns.

PR-AUC — When Positives Are Rare

The Precision-Recall curve fixes this problem by never looking at true negatives at all. It plots precision (y-axis) against recall (x-axis) at every threshold. When the positive class is rare — and in fraud, medical diagnosis, and anomaly detection, it almost always is — the PR curve tells the honest story that the ROC curve sugar-coats.

Here's the key difference. For a random model, the ROC baseline is always 0.5 (the diagonal). For a PR curve, the baseline is the prevalence of the positive class — in our fraud case, 1%. That means even small improvements above 1% are visible and meaningful on the PR curve, while they'd be imperceptible noise on the ROC curve. When positives are rare, the PR curve is a magnifying glass where the ROC curve is a telescope pointed at the wrong galaxy.

Average Precision (AP) summarizes the PR curve as a single number — the weighted mean of precision at each threshold, weighted by the change in recall. It's the go-to metric when the positive class is the one you care about and it's outnumbered.

I'm still developing my intuition for when exactly to switch from ROC to PR. The rough heuristic I use: if the class ratio is worse than 1:10, I look at PR-AUC first and ROC-AUC second. If it's worse than 1:100, I often don't look at ROC-AUC at all.

Log Loss — Punishing Overconfidence

Everything we've talked about so far evaluates hard decisions — did you flag it or not? Did you rank it correctly? But most models output a probability score, and sometimes you care about the quality of that probability itself, not what decision it leads to.

Log loss (also called cross-entropy) evaluates predicted probabilities directly:

log_loss = -(1/N) × Σ[y·log(p) + (1-y)·log(1-p)]

The intuition comes from information theory. Log loss measures surprise. If the model says "99% chance this is legitimate" and it turns out to be fraud, the model is maximally surprised — and log loss punishes it severely. How severely? Let's work through the math. If the model assigns probability 0.01 to the true class, the penalty is -log(0.01) ≈ 4.6. If it assigns 0.5 (complete uncertainty), the penalty is -log(0.5) ≈ 0.69. If it assigns 0.99 (confident and correct), the penalty is -log(0.99) ≈ 0.01.

That asymmetry is the whole point. Being confidently wrong is catastrophically expensive. Being confidently right is nearly free. Being uncertain is somewhere in the middle. This makes log loss a proper scoring rule — a fancy way of saying that the model minimizes its expected loss by reporting its true beliefs. There's no gaming strategy: the best thing the model can do is be honest about its uncertainty.

Log loss connects to something deeper that we'll get to in a moment: the question of whether a model's probability outputs actually mean what they claim to mean.

MCC — The Metric That's Hard to Game

I'll be honest — I didn't pay attention to the Matthews Correlation Coefficient for years. F1 felt like enough. Then I saw a paper (Chicco & Jurman, 2020) that made a persuasive case that MCC is the single most reliable metric for binary classification on imbalanced data, and I started using it. Here's why.

MCC = (TP×TN − FP×FN) / √((TP+FP)(TP+FN)(TN+FP)(TN+FN))

MCC ranges from −1 (every prediction is wrong) through 0 (no better than random) to +1 (perfect). The critical difference from F1: MCC uses all four cells of the confusion matrix. F1 ignores true negatives entirely. That means a model that predicts all negatives gets F1 = 0 (which correctly signals "useless for the positive class") but also MCC = 0 (which correctly signals "useless overall"). So far, same story. But consider a model that makes some predictions but is subtly biased — say it over-flags positives to inflate recall. F1 might be reasonable, but MCC will tank because the product FP×FN in the numerator grows, dragging the score down. MCC treats both classes symmetrically. A high MCC is hard to achieve without genuinely performing well on both positives and negatives.

If you can only report one number on an imbalanced binary problem — one number to a VP who won't read a paragraph — make it MCC.

from sklearn.metrics import matthews_corrcoef
print(f"MCC: {matthews_corrcoef(y_true, y_pred):.3f}")

Calibration — When Probabilities Need to Mean Something

ROC-AUC, PR-AUC, and even F1 all measure ranking or decision quality. They answer: "Does the model score positives higher than negatives?" or "Does it make good decisions at this threshold?" None of them care whether the model's probability outputs are truthful. A model that says "92% chance of fraud" when the real probability is 40% will still rank well — it scores fraud higher than non-fraud — but that 92% number is a lie.

When does the lie matter? When downstream decisions use the probability value itself. Insurance pricing: the premium depends on the predicted risk probability. Medical testing: whether to order an expensive follow-up depends on the predicted probability of disease. Ad bidding: the bid amount is literally a function of the predicted click probability. In all these cases, you need calibration — the property that when a model says "80% likely," it's correct about 80% of the time.

Think of a weather forecaster. A well-calibrated forecaster's "70% chance of rain" predictions should, over time, result in actual rain about 70% of the time. If it rains 95% of the time the forecaster says 70%, the forecaster is underconfident. If it rains only 30% of the time, overconfident. A reliability diagram checks this visually: bin the predicted probabilities into buckets (0–10%, 10–20%, etc.), plot the average prediction against the observed frequency. Perfect calibration falls on the diagonal.

What if your model isn't calibrated? Two post-hoc fixes are standard. Platt scaling fits a logistic regression on the model's raw scores — it works well when the miscalibration is roughly sigmoid-shaped (common for SVMs and neural nets). Isotonic regression fits a more flexible piecewise monotonic function — it handles weirder miscalibration patterns but needs more data to avoid overfitting. Both require a held-out calibration set. Neither changes the model itself — they're a wrapper around its output.

We'll come back to our weather forecaster analogy one more time when we talk about what breaks in production. For now, the takeaway: if you need the model's probability to be an actual probability and not a score, check calibration. Most models out of the box are not calibrated.

Multi-Class Metrics

Everything above was binary — two classes, one confusion matrix. With K classes, you get a K×K confusion matrix and per-class precision, recall, and F1. The question is how to roll those per-class numbers into a single number. Three approaches, each with different politics:

The politics are real. Imagine a wildlife camera that classifies animals: 10,000 deer, 500 coyotes, 12 mountain lions. Macro averaging says the mountain lion class is equally important as the deer class — miss all 12 mountain lions and macro F1 collapses, even if you're perfect on deer and coyotes. Micro averaging would barely notice the mountain lions are gone. Weighted is the middle ground. The right choice depends on who's affected when the model is wrong.

from sklearn.metrics import f1_score

y_true_mc = [0, 0, 0, 1, 1, 2, 2, 2, 2, 2]
y_pred_mc = [0, 0, 1, 1, 0, 2, 2, 2, 1, 0]

print(f"Macro F1:    {f1_score(y_true_mc, y_pred_mc, average='macro'):.3f}")
print(f"Micro F1:    {f1_score(y_true_mc, y_pred_mc, average='micro'):.3f}")
print(f"Weighted F1: {f1_score(y_true_mc, y_pred_mc, average='weighted'):.3f}")

The Regression Scoreboard

Classification asks "which bucket?" Regression asks "how far off?" Different game, different scoreboard. Let's extend our running scenario: alongside the fraud classifier, the bank also predicts expected transaction amounts for anomaly detection. When the predicted amount is way off from the actual, it's a signal worth investigating.

MAE — Mean Absolute Error

MAE = (1/N) × Σ|y_i − ŷ_i|

The simplest regression metric. If we predict a transaction is $300 and it's actually $320, the error is $20. MAE averages all these absolute errors across the dataset. A $10 miss counts exactly twice as much as a $5 miss — no more, no less. You can tell a non-technical person "on average, our predictions are off by $X" and they immediately get it. MAE is robust to outliers because it doesn't amplify anything — one wildly wrong prediction is penalized linearly, same as any other.

RMSE — When Big Misses Are Catastrophic

MSE = (1/N) × Σ(y_i − ŷ_i)²    RMSE = √MSE

Squaring the errors before averaging does something important: it disproportionately punishes large errors. A $100 miss contributes 100× more to MSE than a $10 miss (because 100² / 10² = 100). RMSE takes the square root to get back to the original units — dollars, not dollars-squared — so it stays interpretable.

When is this disproportionate punishment desirable? When a large error isn't "proportionally bad" but catastrophically bad. Energy demand forecasting: underestimate peak load and the grid goes down. Insurance pricing: underestimate risk by 10× and you lose millions. Structural engineering: enough said. In these domains, one big miss is worse than many small misses, and RMSE reflects that. Use MAE when all errors are created equal. Use RMSE when big misses keep you up at night.

R² — Proportion of Variance Explained

R² = 1 − (SS_res / SS_tot)

R² answers a different question: "How much of the variance in the target did my model capture?" R² = 1.0 means the model explains everything — predictions match reality perfectly. R² = 0 means you're no better than predicting the mean every time — your model captured zero signal. R² < 0 means you're actively worse than the mean. That last one surprises people, but it's real, and it means something went fundamentally wrong — wrong features, wrong model, or a bug in preprocessing.

The gotcha: R² always increases (or stays flat) when you add more features, even useless random noise. The model has more knobs and will always find a way to fit the noise a tiny bit. Adjusted R² penalizes for feature count — it drops if a new feature doesn't pull its weight. When comparing models with different numbers of features, always use adjusted R².

MAPE — The Executive Metric

MAPE = (1/N) × Σ|y_i − ŷ_i| / |y_i| × 100%

"Our predictions are off by 5% on average." That's a sentence a VP actually understands. That's MAPE's superpower. Its kryptonite: when actual values are near zero, you divide by something tiny and MAPE rockets to infinity. A transaction of $0.50 with a $2.00 prediction gives a 300% error on a single point. Don't use MAPE for data that crosses zero or hovers near it.

from sklearn.metrics import (mean_absolute_error, mean_squared_error,
                             r2_score, mean_absolute_percentage_error)

y_true_reg = np.array([300, 450, 200, 520, 380])
y_pred_reg = np.array([310, 430, 215, 490, 395])

print(f"MAE:   ${mean_absolute_error(y_true_reg, y_pred_reg):.0f}K")
print(f"RMSE:  ${np.sqrt(mean_squared_error(y_true_reg, y_pred_reg)):.0f}K")
print(f"R²:    {r2_score(y_true_reg, y_pred_reg):.3f}")
print(f"MAPE:  {mean_absolute_percentage_error(y_true_reg, y_pred_reg)*100:.1f}%")

Ranking Metrics — When Order Is Everything

Not everything is classification or regression. Search engines, recommendation systems, and ad platforms care about order — did the best results show up near the top? If you've ever typed a query and the right answer was buried on page three, you've felt the pain that ranking metrics try to quantify.

Let's adjust our running scenario. The bank is building a transaction monitoring dashboard. When an analyst opens the dashboard, they see a ranked list of suspicious transactions, most suspicious first. They have time to investigate maybe 10 per day. The ranking metric tells us: are the actual frauds concentrated at the top of the list, where they'll actually get seen?

MRR (Mean Reciprocal Rank) asks: where does the first real fraud appear in the list? If it's position 1, the score for that query is 1. Position 3? Score of 1/3. Position 10? Score of 1/10. Average across all queries to get MRR. It's the right metric when you have one clear target — navigational search ("find me the homepage of X"), question-answering ("what's the capital of Y"). Its limitation is that it ignores everything after the first hit. If positions 2 through 10 are also frauds, MRR doesn't notice.

MAP (Mean Average Precision) fixes that by considering all relevant results and where they appear. For each position where a fraud appears, compute precision at that position, then average those values. This rewards putting all the frauds at the top and penalizes scattering them randomly through the list. MAP uses binary relevance — a transaction is either fraud or it isn't.

NDCG (Normalized Discounted Cumulative Gain) goes further by handling graded relevance. Not all frauds are equally severe — a $50,000 fraud is more important than a $50 fraud. NDCG lets you assign relevance scores (say, 0 = legit, 1 = suspicious, 2 = likely fraud, 3 = confirmed fraud), discounts the gain of each result by its position (using a logarithmic discount — top positions count far more), and normalizes by the ideal ordering. NDCG@10 is the industry standard for search quality. It ranges from 0 to 1, where 1 means the ranking is perfect.

The one-liner summary: MRR asks "did you find THE answer?", MAP asks "did you find ALL the answers and put them first?", NDCG asks "did you put the BEST answers highest?"

The Reference Table

This is the table I keep pinned above my desk. It's not for learning — it's for deciding.

When Metrics Go Wrong in Production

Everything above assumes that the metric you optimize offline is the metric that matters in the real world. I wish that were true. It usually isn't, and I've learned this the hard way more than once.

There's a saying from economics that captures this perfectly: "When a measure becomes a target, it ceases to be a good measure." That's Goodhart's Law, and it haunts production ML systems. Optimize a recommendation system for click-through rate, and it learns to surface clickbait — engagement goes up, user satisfaction goes down. Optimize a content moderation system for precision, and it becomes so conservative that harmful content flows freely — precision is high, harm is high.

The gap between offline metrics and real-world outcomes shows up in three ways. First, distribution shift: your test set represents the past, but production data represents the future, and the future is different. Second, feedback loops: a fraud model that catches fraud changes the behavior of fraudsters, who adapt, which changes the data the model sees next. Your offline PR-AUC was measured on yesterday's fraudsters, not tomorrow's. Third, proxy misalignment: the metric you can measure (predicted probability, F1 on historical data) is a proxy for the thing you actually care about (dollars saved, lives improved, user trust). Sometimes the proxy diverges from the thing itself.

Remember the weather forecaster? A well-calibrated forecaster's probabilities are honest. But even a perfectly calibrated model can be optimized in a way that makes it useless for the actual decision at hand — because the metric it was trained on isn't the metric the decision-maker needs.

The practical defense is layered. Use offline metrics as a first filter — they're cheap and fast. Use A/B tests to measure what actually happens with real users. Monitor guardrail metrics alongside your primary metric: if click-through rate goes up but session length drops, something's wrong. And when offline and online metrics disagree — and they will — trust the online ones. The lab is not the field.

Wrap-Up

If you're still with me, thank you. I hope it was worth it.

We started with four cells in a confusion matrix and the unsettling discovery that accuracy lies when classes are imbalanced. We built precision and recall as two opposing philosophies — the courtroom versus the cancer ward — and watched them fight over the smoke detector's sensitivity dial. We compressed that tradeoff into F1, learned when to break the symmetry with F-beta, and then stepped back from individual thresholds to see the full picture with ROC and PR curves. We confronted what it means for a model's probabilities to be honest (calibration), met the metric that's hardest to game (MCC), crossed into regression's different scoreboard (MAE, RMSE, R², MAPE), measured the quality of ranked lists (MRR, MAP, NDCG), and ended with the humbling realization that the best offline metric is still a proxy for what actually matters in the real world.

My hope is that the next time someone shows you a model with 99% accuracy and a proud smile, instead of nodding along, you'll ask "on what class distribution?" and "what's the cost when it's wrong?" — and you'll have a pretty darn good mental model of which number to look at instead, and why.

Resources

A curated set of things that helped me understand evaluation metrics at a deeper level:

• Saito & Rehmsmeier (2015) — "The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets." The paper that should have settled the ROC-vs-PR debate years ago. Wildly helpful.
• Chicco & Jurman (2020) — "The advantages of the Matthews correlation coefficient over F1 score and accuracy in binary classification evaluation." The persuasive case for MCC. Changed how I report results.
• scikit-learn metrics documentation — The reference implementations, clearly explained, with the right formulas and edge cases handled. If you ever wonder "what exactly does sklearn compute?" start here.
• Niculescu-Mizil & Caruana (2005) — "Predicting Good Probabilities With Supervised Learning." The O.G. paper on calibration. Shows which models are naturally calibrated and which aren't.
• Google's Rules of ML: Rule #2 — "First, design and implement metrics." A short, unforgettable essay on why choosing the right metric is the single most important decision in a production ML project.

What You Should Now Be Able To Do