A quarter of 36 published malware papers made questionable dataset assumptions; 71 percent described no safety precautions at all. Reproducibility failures in this field are not hypothetical -- they are documented, at scale.
Sign in to mark this article as read and track your progress.
A finding is reproducible if another researcher, using the same data and code, gets the same result. It is replicable if another researcher, collecting fresh data under the same method, gets a consistent result. A literature review should note, wherever the information is available, whether a paper's dataset and code are public. A paper with no available artefacts is harder to build on with any confidence, whatever it claims to have found.
Christian Rossow and seven co-authors examined 36 academic papers, published between 2006 and 2011, that relied on running live malware to collect their data. They found that a quarter of these papers made questionable assumptions about the datasets they used, and that 71% of them gave no description at all of the safety precautions taken while running live malicious code — information any later researcher would need to judge whether the experiment could be safely repeated. These shortcomings turned up in top-tier security venues just as often as in lesser-known ones.
Their paper is now one of the standard references for how a malware experiment should be designed and reported, and it's worth returning to for a specific reason: it shows that the reproducibility problems this course is teaching you to avoid aren't a hypothetical concern from some other field. They're documented, at scale, inside the exact kind of research a malware analyst is trained to do. Neither figure — the 25% or the 71% — is a reason to distrust malware research as a field. Both are a reason to write your own methodology section more carefully than the papers you're reading.
Rossow's paper was about running malware by hand. Pendlebury and colleagues, publishing at USENIX Security in 2019, showed the identical problem had simply moved into machine learning. Android malware classifiers were routinely reported with F1 scores as high as 0.99, appearing almost perfect — yet the authors showed these figures were commonly inflated by two specific errors in experiment design.
Spatial bias occurs when the mix of malicious and benign samples used for testing doesn't match the mix a classifier would actually see after deployment. Temporal bias occurs when a model is trained on samples from one time period and tested on samples that, in a real deployment, would only have existed later — letting the model see the future during training. Once these two biases were removed, several published classifiers' accuracy dropped substantially.
The lesson carries forward directly into how a study's data collection gets designed: a headline accuracy figure is only as trustworthy as the train/test split that produced it, and that split is exactly the kind of methodological detail Rossow's findings already showed the field is prone to under-reporting.
The same problem shows up whenever two independent teams claim to test against "the same" malware family and get different detection results. Often the mismatch traces back to different sample sets, different sandbox configurations, or simply different vendors' labels for what counts as belonging to that family. When you write up your own dataset later in a project, record precisely how each sample was obtained and labelled — not only how many samples there were.
Using the Rossow et al. findings, explain why a review of malware research should record whether a study's dataset and code are public, not only how many samples were used. Then, using the TESSERACT findings, explain the difference between spatial bias and temporal bias in a malware classifier's reported accuracy, and why a very high F1 score alone doesn't rule out either one.
