The efficiency of simulation-based multiple comparisons.

Abstract

A frequently encountered problem in practice is that of simultaneous interval estimation of p linear combinations of a parameter beta in the setting of (or equivalent to) a univariate linear model. This problem has been solved adequately only in a few settings when the covariance matrix of the estimator is diagonal; in other cases, conservative solutions can be obtained by the methods of Scheffé, Bonferroni, or Sidák (1967, Journal of the American Statistical Association 62, 626-633). Here we investigate the efficiency of using a simulated critical point for exact intervals, which has been suggested before but never put to serious test. We find the simulation-based method to be completely reliable and essentially exact. Sample size savings are substantial (in our settings): 3-19% over the Sidák method, 4-37% over the Bonferroni method, and 27-33% over the Scheffé method. We illustrate the efficiency and flexibility of the simulation-based method with case studies in physiology and marine ecology.