Contents

Multiplicity StatisticsEdit

Multiplicity statistics addresses the challenges that arise when a researcher conducts many statistical tests at once. When dozens, hundreds, or millions of hypotheses are evaluated in a single study or across a research program, the chance of stumbling upon at least one apparently significant result purely by chance increases. This has led to a specialized set of methods and best practices designed to control error rates, preserve the integrity of findings, and delineate when a signal is likely real from when it is a false positive.

In practice, multiplicity statistics sits at the intersection of reliability and discovery. On one hand, researchers want to avoid false positives that mislead policy, medicine, or public understanding. On the other hand, overly conservative procedures can suppress legitimate discoveries, especially in fields that rely on large-scale screening or exploratory analysis. The field has therefore evolved toward strategies that balance risk: preventing spurious results while maintaining enough power to detect true effects.

Foundations

  • The core problem is framed as multiple hypothesis testing in the broad sense of hypothesis testing. When many tests are performed, the probability of making one or more Type I errors (false positives) compounds. See hypothesis testing for the general framework.
  • A central distinction is how errors are quantified and controlled. Common targets include the familywise error rate (FWER), the probability of making at least one false positive among all tests, and the false discovery rate (FDR), the expected proportion of false positives among the rejected hypotheses. See familywise error rate and false discovery rate.
  • The p-value remains a common summary statistic, but its interpretation changes in the multiplicity context. Adjusted p-values or adjusted decision rules are often required to maintain intended error control. See p-value.
  • Population structure and relatedness can complicate multiplicity decisions in fields like genomics. Methods that account for population stratification or mixed models help prevent spurious findings from confounding. See population stratification.
  • When decisions must be made about many features (genes, SNPs, behavioral measures), efficiency and interpretability matter. This has spurred a spectrum of approaches from strict FWER control to more permissive FDR-based strategies, each with practical trade-offs. See Benjamini-Hochberg procedure and Bonferroni correction.

Methods for controlling multiplicity

  • Bonferroni correction: A simple, conservative approach that divides the desired error rate by the number of tests. It is widely used for its transparency and ease of interpretation. See Bonferroni correction.
  • Holm-Bonferroni method: A stepwise improvement over the classic Bonferroni, offering more power while maintaining FWER control. See Holm-Bonferroni.
  • Hochberg's step-up procedure: Another stepwise method that can be more powerful than traditional Bonferroni in certain settings. See Holm-Bonferroni (discusses related stepwise methods).
  • False discovery rate control (FDR): A shift in perspective from avoiding any false positives to controlling the expected proportion of false positives among positives. This is particularly useful in large-scale screening. See false discovery rate and the Benjamini-Hochberg procedure.
  • Benjamini-Hochberg procedure: The canonical FDR-controlling method for ordered p-values, widely adopted in genomics and other data-rich fields. See Benjamini-Hochberg procedure.
  • Storey methods and q-values: Extensions that estimate the proportion of true null hypotheses and provide direct measures (q-values) for decision-making. See q-value.
  • Hierarchical and gatekeeping procedures: Strategies that structure testing across families of hypotheses or endpoints, preserving power where it matters most while maintaining overall error control. See hierarchical testing (often discussed in the context of multiple endpoints).
  • Bayesian and empirical-Bayes approaches: An alternative to classical (frequentist) error control, using prior information and probability models to manage multiplicity. See Bayesian statistics and empirical Bayes.

Applications

  • Clinical trials and regulatory science: In medical research, multiplicity control is crucial when multiple endpoints or subgroups are analyzed, or when multiple dose groups are compared. Stricter FWER-like controls are common in high-stakes settings, while FDR-like approaches appear in exploratory phases. See clinical trial.
  • Genomics and high-throughput screening: Large-scale screens test thousands to millions of features, making FDR-based strategies a practical default to maintain discovery potential. See Benjamini-Hochberg procedure.
  • Psychology and social science: Replication and robust significance testing have brought multiplicity considerations to the forefront, with preregistration and emphasis on power and transparency. See preregistration.
  • Environmental and climate science: Large observational datasets and multiple model comparisons raise multiplicity questions, prompting careful reporting of adjusted results. See hypothesis testing.
  • Population studies and epidemiology: Adjustments for correlated tests and population structure are common, with methods that handle dependence among tests. See population stratification.

Controversies and debates

  • Power versus conservatism: A perennial debate centers on the balance between avoiding false positives (high reliability) and maintaining the ability to detect real effects (statistical power). In fields with many tests, FDR methods are attractive for preserving discovery potential, but some researchers argue that important findings can still be missed if corrections are overly aggressive. See discussions of Benjamini-Hochberg procedure and Bonferroni correction.
  • Dependence and test structure: Real-world data often involve correlated tests. Procedures that assume independence can become anti- or over-conservative when dependence is strong. This has driven development of methods that accommodate dependence but adds complexity. See literature around multiple testing and hierarchical testing.
  • Bayesian versus frequentist viewpoints: Some researchers favor Bayesian or empirical-Bayes frameworks as a way to incorporate prior knowledge and model uncertainty, potentially handling multiplicity without explicit error-rate targets. Critics argue that priors can be subjective and reintroduce bias if not chosen carefully. See Bayesian statistics and empirical Bayes.
  • Reproducibility and the replication crisis: Critics of scientific practice have pointed to unreliability in published findings, often highlighting misuse of p-values or selective reporting. Proponents of multiplicity controls argue that transparent reporting, preregistration, and replication are essential complements to statistical corrections. See reproducibility and preregistration.
  • Ideological criticisms and methodological norms: In broader cultural debates about science and policy, some critics argue that strict statistical norms serve as a political gatekeeping mechanism. Proponents respond that reliability in evidence matters across disciplines and political perspectives, and that robust methods reduce the risk of guiding policy by noise rather than signal. While discussions can become heated, the core issues are about risk management, incentives, and the practical consequences of false positives versus missed discoveries.

From a pragmatic standpoint, the multiplicity statistics toolkit should reflect the stakes of the decision. In high-stakes contexts like drug approvals or life-saving interventions, minimizing false positives through controls like FWER is often warranted. In exploratory research and large-scale screening, methods that control the false discovery rate provide a means to keep doors open for genuine discoveries while still policing the rate of false leads. The ongoing debates reflect the tension between reliability and discovery, and they tend to be resolved not by ideology but by aligning method choice with the consequences of errors and the structure of the data.

See also