Scope and Limitations

Stating What You Don't Claim

Every study needs to state what it does not claim. This is not an apology — a study with no stated boundary is one whose claims cannot be checked by anybody, including the person who ran it.

Scope is the boundary the researcher chose on purpose: which dataset, which population, which time window, which threat model, which metric.

Limitations are constraints the researcher did not choose but must disclose anyway: a small sample, an imbalanced dataset, a single-vendor test environment, or simply not enough time to run a larger study.

A limitations section written properly is a service to the next researcher: it says exactly which follow-up study would extend the finding, and it stops the current claim from being read more broadly than the evidence actually supports. A malware classifier that reaches 99% accuracy on one organisation's endpoint logs from a single quarter has a result worth reporting — provided the report says, in the same breath, that the figure hasn't yet been checked against a different organisation, a different malware family mix, or a different time period.

Further Reading

• M. Antonakakis et al., "Understanding the Mirai Botnet," 26th USENIX Security Symposium, 2017, pp. 1093–1110.
• C. Rossow et al., "Prudent Practices for Designing Malware Experiments: Status Quo and Outlook," IEEE Symposium on Security and Privacy, 2012, pp. 65–79 — a paper this course returns to when it covers research design in Unit 2.
• Reporting on the WannaCry outbreak and the discovery of its kill switch, May 2017 — widely covered by Microsoft's own security blog and by independent researchers including Marcus Hutchins.

Check Your Understanding

A student claims a malware classifier is "99% accurate" after testing it on 200 samples collected from one antivirus vendor's public feed. Write a scope statement and a limitations statement for this claim.