Contents

Multiple Hypothesis TestingEdit

Multiple hypothesis testing arises whenever a study evaluates more than one hypothesis at once. In data-rich environments, researchers routinely test many possibilities in parallel—from scanning genomes for disease-associated variants to conducting dozens or hundreds of variants of an internet experiment. The central challenge is that testing multiple hypotheses inflates the chance of false positives unless adjustments are made. In practice, this has a direct bearing on the credibility of findings, the allocation of public research funds, and the reliability of conclusions that inform policy and business decisions.

As a result, statisticians distinguish between making inferences about a single hypothesis and controlling error rates across a family of tests. The history of the field includes a crescendo of methods designed to guard against false positives while preserving legitimate discoveries. The tension between avoiding spurious results and maintaining power to detect real effects is especially acute when the number of tests is large, when data are messy, or when results feed into high-stakes decisions such as medical approvals or regulatory policy. See also Null hypothesis significance testing and Statistical power for related ideas.

Fundamentals

  • Familywise error rate (FWER): the probability of making at least one false positive among all tests in a family. Controlling the FWER tends to be conservative, especially as the number of tests grows.
  • False discovery rate (FDR): the expected proportion of false positives among the tests declared significant. Controlling the FDR is typically less conservative and more powerful in large-scale testing settings.
  • Type I error: rejecting a true null hypothesis (a false positive). In multiple testing, the risk of Type I error compounds across tests.
  • Statistical power: the probability of detecting a true effect. There is a trade-off between reducing false positives and maintaining power to discover real effects.

Common methods and ideas - Bonferroni correction: adjusts the per-test significance level by dividing the target alpha by the number of tests. Simple and robust but often very conservative. - Holm-Bonferroni method: a step-down procedure that is uniformly more powerful than the basic Bonferroni while still controlling the FWER. - Hochberg procedure: a step-up version that can offer even greater power under certain conditions. - Benjamini-Hochberg procedure (BH): a widely used method to control the FDR, especially in high-throughput contexts like genomics and neuroimaging. - Storey and Tibshirani q-values: practical approaches to estimating the false discovery proportion and adjusting significance thresholds accordingly. - Permutation-based and resampling methods: data-driven ways to estimate error rates under the observed correlation structure.

Link these concepts as you discuss them: Bonferroni correction, familywise error rate, False discovery rate, Benjamini–Hochberg procedure, p-value, Statistical power, Preregistration, and Data dredging.

Methods and practice

  • When the goal is to minimize any false positive findings (e.g., in high-stakes regulatory settings), controlling the FWER via methods like the Holm-Bonferroni or classic Bonferroni correction can be appropriate.
  • When the research aims to identify a set of potentially interesting findings for further study, and some false positives are acceptable in the short term, controlling the False discovery rate is often preferred, with BH as a standard tool.
  • In large-scale testing, dependence among tests matters. Some methods assume independence; others accommodate certain dependency structures through permutation or modeling approaches. See Permutation test and Dependency considerations for details.
  • Transparency in reporting is essential. Distinguishing confirmatory tests (preregistered and hypothesis-driven) from exploratory analyses helps readers interpret results accurately. See Preregistration and Data dredging for related concerns.

Applications and domains - Genomics and high-throughput biology: many variants are tested for association with a trait, making FDR control a practical compromise between discovery and reliability. - Neuroimaging: voxel-wise testing across the brain produces thousands of tests, motivating false discovery control and reporting of effect sizes. - A/B testing and product experimentation: multiple metrics and segments can yield numerous tests; proper adjustment protects decision-making from being swayed by noise. - Social sciences and economics: debate continues over how strictly to correct for multiple testing given the risk of discarding meaningful effects.

Controversies and debates

  • The proper balance between conservatism and discovery: supporters of strict FWER control argue that it protects against overinterpretation, while critics contend that excessive conservatism suppresses genuine findings, especially in exploratory science. From a practical standpoint, many laboratories and journals favor FDR-based approaches for large-scale testing because they preserve power.
  • P-values and practical significance: a test may be statistically significant after adjustment but yield an effect that is negligible in real-world terms. Critics urge emphasis on effect sizes and confidence intervals in addition to p-values. See P-value and Effect size for context.
  • Preregistration versus exploration: preregistration reduces the risk of data dredging and p-hacking by committing to an analysis plan before data are seen. Proponents argue this enhances credibility; critics, particularly in fast-moving fields or proprietary contexts, warn it can hinder legitimate exploratory work and slow progress. See Preregistration and P-hacking.
  • Dependence and real-world data: many testing settings involve correlated tests (e.g., related outcomes, related genomic markers). Some critique standard methods for not fully accounting for complex dependencies, while others advocate permutation-based or model-based solutions that adapt to the observed data structure.
  • Policy and regulation implications: in government and industry, the choice of error-rate control can influence which findings inform policy. Conservative defaults reduce the risk of acting on false positives but may delay or suppress beneficial interventions. Advocates of robust standards argue that taxpayers and consumers deserve reliable evidence before policy changes; critics warn against enabling a stifling environment that slows innovation.

From a practical, results-oriented perspective, the debate often centers on whether the goal is to prevent any false positives at the expense of missing true effects, or to allow more discoveries with a controlled but higher false-positive rate. In policy circles, the preference tends to tilt toward methods that yield reproducible conclusions and transparent reporting, while in fast-paced industry settings there is pressure to balance speed with statistical integrity.

Applications and examples

  • Genome-wide association studies (GWAS) routinely apply FDR-based criteria or permutation-derived thresholds to cope with millions of tests. See Genome-wide association study.
  • Neuroimaging studies use correction schemes to account for thousands of voxels tested across the brain; reporting often pairs adjusted p-values with effect sizes. See Neuroimaging.
  • Clinical trials and meta-analyses must decide how to handle multiple endpoints and subgroup analyses, balancing the risk of spurious claims with the need to identify meaningful therapeutic effects. See Clinical trial and Meta-analysis.
  • In industry, large-scale A/B testing requires careful control of erroneous conclusions when decisions affect products and revenue; practitioners may preregister primary endpoints while exploring secondary outcomes with clear labeling. See A/B testing.

See also