A 95% accurate research classifier can miss every useful candidate

A classifier can correctly label 95 out of 100 candidates and still find none of the candidates you actually want. The first question for a research screening task is therefore more specific than “Is the accuracy high?”: which cases did the system find, and which did it miss?

Here is a complete synthetic example you can check without training a model or calling an AI service. Think of a research workflow that screens experimental conditions for a useful property. In this example, label 1 means the condition has that property and label 0 means it does not. These labels and predictions are invented teaching data. They are not experimental measurements or performance results from an actual AI model.

Keep the test denominator fixed

Our test set has 100 records: 5 positive and 95 negative. The first prediction vector labels all 100 records negative.

This gives 95 correct predictions out of 100, so accuracy is 95%. But the system finds zero of the five positive candidates. Positive-class recall is 0 / 5 = 0%.

Precision asks a different question: among the candidates reported as positive, how many are actually positive? This system reports no positive candidates, so the denominator is zero. The mathematical ratio is undefined. A software library may expose a configurable convention for zero division; a report should state that convention rather than silently treating the result as perfect precision.

A lower accuracy can accompany fewer missed candidates

Now keep exactly the same records and actual labels. Construct a second prediction vector that identifies four of the five positive candidates, but also reports six negative candidates as positive.

The second vector gives:

• Precision: 4 / (4 + 6) = 40%.
• Recall: 4 / (4 + 1) = 80%.
• Accuracy: (4 + 89) / 100 = 93%.

Accuracy fell from 95% to 93%. Missed positive candidates fell from five to one, while false positives rose from zero to six. Those are different changes, with different research costs. Missing a promising condition may be expensive; so may spending scarce laboratory resources on false positives. These counts alone do not establish that the second prediction vector is preferable.

We constructed both vectors directly. This is not evidence that threshold tuning improved a trained model, or that a particular model can achieve these results in a laboratory.

Reproduce the arithmetic with standard Python

The code below keeps record identifiers, labels and predictions aligned. It does not use an external dataset, model, network request or third-party package.

from collections import Counter ids = [f"test-{i:03}" for i in range(100)] actual = [1] * 5 + [0] * 95 all_negative = [0] * 100 alternative = [1] * 4 + [0] + [1] * 6 + [0] * 89 def audit(record_ids, labels, predictions): assert len(record_ids) == len(labels) == len(predictions) assert len(record_ids) == len(set(record_ids)) assert len(labels) > 0 assert set(labels) <= {0, 1} assert set(predictions) <= {0, 1} cells = Counter(zip(labels, predictions)) tp, fn = cells[1, 1], cells[1, 0] fp, tn = cells[0, 1], cells[0, 0] def ratio(numerator, denominator): return numerator / denominator if denominator else None return { "TP": tp, "FP": fp, "TN": tn, "FN": fn, "test_n": len(labels), "accuracy": ratio(tp + tn, len(labels)), "precision": ratio(tp, tp + fp), "recall": ratio(tp, tp + fn), } print(audit(ids, actual, all_negative)) print(audit(ids, actual, alternative)) 

The first output is TP=0, FP=0, TN=95, FN=5, with accuracy 0.95, precision None and recall 0.0. The second is TP=4, FP=6, TN=89, FN=1, with accuracy 0.93, precision 0.4 and recall 0.8.

The duplicate-identifier check catches one simple input problem. It does not establish independence from training data, correct labeling, representative sampling or freedom from preprocessing leakage. Those require separate evidence about how the evaluation set and predictions were produced.

Give an AI assistant a bounded audit task

An assistant can help prepare this table, but it needs enough information to preserve the denominator. A usable request is:

Positive label 1 means the experimental condition has the target property. Using only the attached per-record labels, predictions and test identifiers, report TP, FP, TN and FN. Show the numerator and denominator for accuracy, positive-class precision and positive-class recall. Report zero denominators explicitly. Keep the two prediction versions separate. Flag duplicate identifiers or missing labels; do not silently drop rows or invent replacements. Do not train a model or choose a threshold from this test set.

For actual research, retain the prediction version, label definition, split unit and evaluation-set selection rules. If a classification threshold needs tuning, choose it within an appropriate validation or cross-validation process before final independent testing. Repeatedly selecting the threshold that looks best on the final test set changes what that test can tell you.

The high score in the opening example was arithmetically correct. The missing information was which cases that score counted as success.

Sources, read November 3, 2026:

• scikit-learn 1.9.1: classification metrics, including accuracy, precision, recall and imbalanced classification.
• scikit-learn: tuning the decision threshold, including positive-label definitions and validation boundaries.

Written by the RigorMed team. All example labels and predictions above are synthetic teaching data.