This content is open to everyone — sign in to save your progress, earn points, and unlock module exams.
Unit 3 of Research Methodology
Where research data actually comes from and how trustworthy the instrument collecting it is, how to sample a population honestly, how to state and test a hypothesis without overclaiming, what scale of measurement a given number actually is, which statistical test a question calls for, and the tools used to run it.
Learning outcomes
The difference between data you collect yourself and data someone else collected first, and why a secondary dataset's original labels deserve scrutiny before you trust them for a new question.
Questionnaires, interviews, and sandbox telemetry as instruments -- the tradeoff between an instrument's reach and its depth, and why a sandbox that detects itself and produces empty data is still a finding.
Four kinds of validity, three kinds of reliability, and why Unit 2's two biggest reproducibility failures -- Rossow's inconsistent malware labels and TESSERACT's spatial and temporal bias -- are actually reliability and validity problems wearing different names.
Population, sampling frame, and sample -- and why choosing a stratified sample over a simple random one is often the more honest design when a characteristic is expected to vary sharply across subgroups.
Null and alternative hypotheses, significance level and p-value, Type I and Type II error, and why a test can reject a hypothesis but can never prove it true.
Nominal, ordinal, interval, and ratio scales -- and why averaging a low/medium/high/critical severity rating and reporting it as a precise number is one of the most common statistical errors in student projects.
The independent t-test, paired t-test, one-way ANOVA, chi-square test, Pearson correlation, and simple linear regression -- what each one assumes about your data, and a quick decision path for picking between them.
SPSS, R, Python, Excel, and Jupyter compared for security research, a worked paired t-test in Python using scipy, and why a methodology section must name the exact tool and function used, not just the test.
The choices that determine whether an ML experiment's results are trustworthy — dataset splits, cross-validation, ablation studies, baselines, and statistical significance — and the reproducibility practices that let another researcher verify them.
