Identifying important results from multiple statistical tests.

Abstract

When many statistical tests are performed simultaneously, the overall chance of a type I error (incorrect rejection of a true null hypothesis) can substantially exceed the nominal error rate used in each individual test. Numerous techniques exist to adjust results of individual tests to control this problem. In general, these techniques apply a more stringent criterion of statistical significance (a smaller P-value) to each individual test than normally needed to maintain the experimentwise type I error. With an analysis that seeks to identify results for further research, however, such a conservative technique may not be appropriate. We present a new approach that uses a mixture of several distributions to model the set of P-values or of test statistics. One component models the results consistent with a failure to reject the null hypothesis, while the other distribution(s) in the mixture represent results inconsistent with the null hypothesis. These latter results may not achieve statistical significance based on a conventional P-value. We illustrate the use of the method on national mortality data and on several data sets analysed previously.