Reliability is about how tightly your measurements cluster. Validity is about whether that cluster sits on the target. Unit 2's two biggest reproducibility failures turn out to be exactly these problems wearing different names.
Sign in to mark this article as read and track your progress.
Building an instrument is only half the job. The other half is asking whether it can be trusted, and trust here splits into two separate questions.
Face validity. On the surface, does the instrument look like it measures what it claims to? A questionnaire item asking "how often do you change your password" has face validity for a security-hygiene study; asking someone's favourite colour does not.
Content validity. Does the instrument cover the full concept, not just a convenient slice of it? A security-awareness quiz that only asks about phishing, and never about physical security or password hygiene, has weak content validity for "security awareness" as a whole.
Construct validity. Does the instrument actually capture the underlying idea it claims to, as opposed to something merely correlated with it? A malware detector's test accuracy has weak construct validity for "real-world protection" if the test conditions don't resemble how the malware would actually be encountered.
Criterion validity. Does the instrument's result agree with an independent, trusted measure of the same thing? A quick five-question phishing-susceptibility quiz has good criterion validity if people who score poorly on it also click more simulated phishing links in practice.
Test-retest reliability. Does the instrument give a similar result when applied to the same subject twice, with nothing relevant changed in between?
Internal consistency. On a multi-item questionnaire meant to measure one underlying concept, do the items agree with each other? This is usually reported as Cronbach's alpha, a coefficient between 0 and 1; a value of around 0.7 or higher is the conventional minimum most reviewers expect, though this cutoff is a widely used rule of thumb rather than an absolute law, and more critical, applied decisions are usually held to a stricter standard.
Inter-rater reliability. When two or more people independently judge or label the same thing, how often do they agree? Two malware analysts independently labelling the same 100 samples by family, and agreeing on 90 of them, have shown reasonable inter-rater reliability.
Reliability is about how tightly repeated measurements cluster together. Validity is about whether that cluster actually sits on the target. An instrument can be reliable without being valid — consistently wrong is still wrong — and it can be valid in principle while being unreliable in practice, giving a different answer every time you ask.
Unit 2's two big reproducibility cases are, underneath, validity and reliability problems wearing different names. Rossow and colleagues' finding that different papers used inconsistent, vendor-dependent malware family labels is an inter-rater reliability problem: different "raters" (the antivirus vendors) disagreed on the label for the same sample. TESSERACT's spatial and temporal bias is a construct validity problem: a high F1 score wasn't actually capturing "performance in real-world deployment," the construct the researchers cared about, because the test conditions didn't resemble deployment conditions. Naming the specific failure this precisely, rather than saying only "the study had problems," is what a strong methodology section does.
Two independent malware analysts each label the same 50 samples by family. Name the specific type of reliability this checks, and explain why high agreement between the two analysts still wouldn't, by itself, prove either of them is correct.
