The choice of software rarely changes what a test does -- it changes how quickly and how reproducibly the work gets done. A worked paired t-test in Python, and why "we ran a t-test" is not enough for a methodology section.
Sign in to mark this article as read and track your progress.
The choice of software rarely changes what a test does; it changes how quickly and how reproducibly the work gets done.
| Tool | Strengths and typical use |
|---|---|
| SPSS | Menu-driven statistics package, common in survey-based security research; good for standard tests without writing code. |
| R | A language built around statistics; strong for formal hypothesis testing and publication-quality plots. |
| Python | General-purpose, with mature libraries such as pandas and scipy; the natural choice when the same pipeline also needs to train a model or parse log files. |
| Excel | Fast for small datasets and quick pivot tables; weak once a project needs more than basic descriptive statistics. |
| Jupyter | A notebook environment that runs Python or R interactively, mixing code, output, and narrative in one document — useful for keeping an analysis reproducible. |
A comparison of two malware classifiers' accuracy across ten runs each can be tested directly with scipy, given the per-run accuracy scores for both:
from scipy import stats
clf_a = [86, 89, 87, 90, 88, 85, 91, 87, 89, 90]
clf_b = [92, 94, 93, 95, 92, 93, 96, 91, 94, 94]
t_stat, p_value = stats.ttest_rel(clf_a, clf_b)
ttest_rel runs a paired t-test, the correct choice here because both classifiers were scored on the same ten runs rather than on two independent samples. A p-value below 0.05 would support the claim that Classifier B's advantage is unlikely to be run-to-run noise.
Whatever tool is used, the version of the software and the exact function called belong in the methodology section of the eventual report. "We ran a t-test" is not enough. "We ran a paired two-tailed t-test using scipy.stats.ttest_rel" is what lets another researcher check the work — the same "valid and verifiable" test from Unit 1.
Why does "we ran a t-test" fall short of what a methodology section needs, even if the p-value reported is correct?