Reporting 98% accuracy on a test set that the model saw during development is not a result — it is a measurement error. ML research has its own experimental design discipline: how you split data, choose baselines, structure ablations, and report variance determines whether your numbers mean anything.
Sign in to mark this article as read and track your progress.
Running a neural network and reporting the accuracy is not an experiment in the research sense. An experiment is a procedure that isolates the effect of a variable under controlled conditions so that the result is attributable to that variable rather than to confounding factors. ML research is full of results that fail this test — models evaluated on training data, comparisons made against weak baselines, accuracy numbers reported without variance across runs.
This article covers the design decisions that determine whether an ML paper's results are trustworthy.
The most fundamental rule in ML experimental design is that the test set must never be seen during model development. The test set exists to simulate unseen deployment data. If any decision about the model — architecture, hyperparameters, stopping criterion — was influenced by test-set performance, the reported accuracy is optimistic and the experiment is compromised.
The standard split in ML research is three-way:
| Split | Purpose | Typical proportion |
|---|---|---|
| Training | Model learns weights | 60–80% |
| Validation | Hyperparameter tuning, early stopping | 10–20% |
| Test | Final evaluation — touched once | 10–20% |
The validation set is used repeatedly during development. The test set is used once, at the end, to produce the final reported number. Any researcher who reports "I tuned on the test set to find the best threshold" has produced a number that is not comparable to other papers evaluated on genuinely unseen data.
When a dataset is small — a few thousand samples or fewer — a single train/val/test split produces results that are highly sensitive to which samples happened to land in the test set. k-fold cross-validation addresses this by rotating the test set across k equally-sized folds:
The mean is the estimate of generalisation performance; the standard deviation quantifies its uncertainty. A paper that reports "mean F1 = 0.91 ± 0.03 across 10-fold CV" is far more trustworthy than one that reports "F1 = 0.94" on a single split.
Stratified k-fold preserves the class ratio in each fold, which is essential for imbalanced datasets — typical in intrusion detection and malware classification, where attack samples are rare. Without stratification, some folds may contain no attack samples at all, which makes the fold's evaluation meaningless.
A baseline is the performance a new method must beat to demonstrate that it is useful. Choosing the right baseline is an ethical and scientific obligation — a weak baseline makes any result look impressive.
A baseline must satisfy three conditions:
Competence: the baseline must be a reasonable current approach, not a naive rule or a method that has been superseded. Comparing a new intrusion detection model to a 2005 signature-based system may be easy to beat but tells the research community nothing useful.
Fair conditions: the baseline must be trained and evaluated under identical conditions — same data splits, same preprocessing, same evaluation metric. Comparing your model's F1 on balanced data to the baseline's F1 on imbalanced data is not a comparison; it is an artefact.
Reproducibility: if you implement the baseline yourself, report the implementation details. If you use published numbers from the baseline's own paper, note any differences in evaluation conditions.
An ablation study removes one component of a proposed system at a time to measure how much that component contributes to performance. It answers the question: "Does each part of my proposed method actually help, or does the improvement come from one component that could have been used alone?"
A well-designed ablation study for a proposed hybrid malware classifier might test:
If removing the attention layer drops F1 from 0.93 to 0.92 but removing the CNN drops it to 0.71, the ablation shows that the CNN feature extractor is doing most of the work. This is an honest and informative result, even if the attention layer's contribution is modest.
Reporting that Model A achieved 93.4% accuracy and Model B achieved 91.7% on the same test set does not establish that Model A is better — it shows a difference. Whether that difference is replicable or due to randomness in the test set requires a statistical test.
Two tests commonly used in ML research:
McNemar's test compares two classifiers on the same dataset by examining the pattern of disagreements — samples where one classifier is right and the other is wrong. It is appropriate when both classifiers are evaluated on the same fixed test set.
Wilcoxon signed-rank test compares the distributions of performance across multiple runs or cross-validation folds. It is non-parametric, making no assumption about normality, which is appropriate for small numbers of folds.
Report the p-value and the chosen significance threshold (typically α = 0.05). A result described as "significantly better" must be backed by a test — a large raw difference in accuracy is not significance.
An ML experiment is reproducible if another researcher with access to the same data can run the code and obtain the same (or statistically indistinguishable) results. In practice, several factors break reproducibility:
Random seeds: neural network weight initialisation, dataset shuffling, and dropout all use pseudorandom number generators. Report and fix all seeds used in an experiment. Without this, results vary across runs.
Environment: Python version, library versions (PyTorch, scikit-learn, NumPy), CUDA version, and hardware can all affect results. A requirements.txt or environment.yml file pinning exact versions is the minimum; a Docker container or published conda environment is stronger.
Data availability: if the dataset is public, cite its specific version. If it is proprietary, describe its composition and class distribution in enough detail that a comparable dataset can be constructed.
The TESSERACT study (discussed in Unit 3 under validity and reliability) is a concrete example of what happens when experimental design fails: malware classifiers trained and evaluated on data from the same time window appear accurate, but perform poorly when evaluated on data from a later window because they have learned temporal artefacts rather than malware behaviour. Controlling for temporal split — treating time as the split criterion rather than random sampling — is an experimental design choice, not a tuning choice.
A student trains a random forest classifier on a malware dataset, achieving 97% accuracy on the test set. When asked how the test set was selected, the student says: "I tuned the decision threshold on the test set to maximise F1, then reported the accuracy." Identify every experimental design error in this procedure and describe what an honest evaluation protocol would look like.
