A test never proves the null hypothesis true. It only finds enough evidence to reject it, or it doesn't. Getting that distinction right is what separates a defensible result from an overclaimed one.
Sign in to mark this article as read and track your progress.
Unit 1 introduced the hypothesis as a tentative, testable statement. This topic makes that testing precise enough to act on.
The null hypothesis (H₀) is the default position: no effect, no difference, no association. "The mean detection time of the new IDS rule set is the same as the baseline." The alternative hypothesis (H₁) is what the researcher suspects might actually be true. "The mean detection time of the new rule set is different from the baseline."
A test never proves H₀ true. It either finds enough evidence to reject H₀, or it doesn't find enough evidence and fails to reject H₀ — which is a weaker claim than "H₀ is confirmed."
The significance level (α) is the risk of rejecting H₀ when it's actually true, that the researcher accepts in advance. Conventionally set at 0.05, decided before the data are collected, not after.
The p-value is the probability of seeing data at least as extreme as what was actually observed, if H₀ were true. When the p-value falls below α, the result is called statistically significant.
A two-tailed test is used when the alternative hypothesis only claims a difference, in either direction. A one-tailed test is used when the alternative claims a direction, such as "faster than." Choosing one-tailed after already seeing the data, purely to make the p-value smaller, is a well-known way to mislead a reader and should never be done.
A test can go wrong in exactly two ways, and this borrows directly from the confusion matrix used to score a classifier:
| H₀ is actually true | H₀ is actually false | |
|---|---|---|
| Reject H₀ | Type I error (false positive) | Correct decision |
| Fail to reject H₀ | Correct decision | Type II error (false negative) |
This parallel is worth taking seriously rather than treating as a teaching trick. Setting α lower reduces Type I error but raises Type II error, in exactly the same way that tightening an IDS's alert threshold cuts false alarms at the cost of missing more genuine attacks. A researcher choosing α = 0.01 instead of the usual 0.05 is making the same trade-off a SOC manager makes when tuning a detection rule.
A SOC has historically taken an average of 42 seconds to detect a certain attack pattern. After deploying a new detection rule, the team times 25 detections and runs a one-sample test comparing the new average against the historical baseline, at the standard 5% significance level. Suppose the test fails to reach significance. The honest conclusion is not that the new rule made no difference — it's that 25 timed detections weren't enough to be confident, at the 5% level, that any drop was more than ordinary variation. The next step is a larger sample, not a more relaxed significance level chosen after the fact to force a "significant" result.
A p-value only answers "is this likely to be more than chance," and a large enough sample can make even a tiny, practically meaningless difference statistically significant. Effect size answers the question a p-value can't: how big is the difference? Cohen's d, a common measure for comparing two means, is conventionally read as small around 0.2, medium around 0.5, and large around 0.8 — though Cohen himself intended these as a rough last resort, not a rigid rule, and what counts as a meaningful effect in security research depends on what the difference would actually cost or save in practice. Reporting effect size alongside the p-value, not instead of it, is what lets a reader judge whether a statistically significant improvement is also an operationally worthwhile one.
Explain why a hypothesis test can reject H₀ but can never prove H₀ true, and what the difference is between "failing to reject H₀" and "confirming H₀." Then, using the Type I / Type II error table above, explain what it would mean for a research team to set an unusually strict significance level of α = 0.001 instead of the conventional 0.05 — what is the team trading away to get that stricter threshold?
