Most tests a security research project will ever need fall into a small set. The choice turns on two questions: how many groups are being compared, and whether the outcome is continuous or categorical.
Sign in to mark this article as read and track your progress.
Most tests a security research project will ever need fall into a small set, and the choice between them turns on two questions: how many groups are being compared, and whether the outcome is continuous or categorical.
| Test | Data | Typical use |
|---|---|---|
| Independent-samples t-test | Continuous, 2 independent groups | Compare mean response time of two different SOC teams. |
| Paired t-test | Continuous, 2 matched measurements | Compare the same malware samples' detection scores before and after a signature update. |
| One-way ANOVA | Continuous, 3+ independent groups | Compare mean detection accuracy across three antivirus products. |
| Chi-square test of independence | Categorical, 2+ groups | Test whether phishing-click behaviour is associated with department. |
| Pearson correlation | Two continuous variables | Measure the association between patch-cycle length and incident severity. |
| Simple linear regression | One outcome, one or more predictors | Predict incident response time from team size and alert volume. |
Every test assumes something about the data before it can be trusted. Checking an assumption takes a few minutes; reporting a result that quietly violates one doesn't survive peer review.
| Test | What it assumes about your data |
|---|---|
| t-test (either kind) | The outcome is roughly normally distributed within each group; for the independent version, the two groups' variances should be roughly equal. |
| One-way ANOVA | The same normality and equal-variance assumptions, extended to three or more groups. |
| Chi-square test | Each expected cell count should generally be 5 or more; with smaller samples the test becomes unreliable. |
| Pearson correlation | Both variables are continuous and their relationship is roughly linear, not curved. |
Correlation and ANOVA in particular are easier to trust once you've actually looked at the data rather than only at a single summary statistic. Plotting patch-cycle length against incident severity for each organisation, before running a correlation test, is how you catch a curved relationship that Pearson correlation would badly misjudge. Plotting the spread of response times for each antivirus product side by side, before running an ANOVA, shows whether the group differences look larger than the normal run-to-run variation within each group.
For each scenario below, name the test you'd use:
Answers: (1) paired t-test, (2) one-way ANOVA, (3) Pearson correlation, (4) chi-square test of independence.
A quick first-pass guide: if the outcome is categorical, use chi-square. If the outcome is continuous and there are only two groups, use a t-test — paired if the same subjects were measured twice, independent if not. If the outcome is continuous and there are three or more independent groups, use one-way ANOVA. Paired data, correlation, and regression sit outside this simple flow, and are chosen when the question itself asks for an association or a prediction rather than a group comparison.
A phishing simulation is sent to 200 employees, 50 from IT and 150 from the rest of the organisation. Far fewer of the IT staff click the simulated link than the company-wide rate would predict. A chi-square test of independence on the resulting two-by-two table — department against clicked-or-not — is the right tool to ask whether that gap is larger than chance would produce. If the test statistic exceeds the critical value for one degree of freedom at the 5% level, the researcher can report that department and click behaviour are associated in this dataset, with IT employees clicking noticeably less often.
Which test would you use to check whether larger malware files tend to make more API calls during execution, and what result would count as support for that claim? Then: a department head argues that because only 10% of IT staff clicked the simulated phishing link, compared with 23% company-wide, no formal test is needed to prove the difference is real. Explain why a chi-square test is still the correct next step.
