A comparison of multivariable mathematical methods for predicting survival--III. Accuracy of predictions in generating and challenge sets.
This paper concludes a study of "performance variability" when four methods of multivariable analysis--multiple linear regression, discriminant function analysis, multiple logistic regression, and two arrangements of Cox's proportional hazards regression--were applied to the same stratified random samples of "generating sets" containing seven different statistical distributions of cogent biologic attributes in a composite staging system for a large cohort of patients with lung cancer. Each model developed from the generating sets was also applied for predictions in a previously sequestered "challenge set". Across the different generating sets, the multivariable methods showed good agreement with one another in the stepwise choice of first two powerful predictor variables, but not in the sequence of subsequent choices or in the standardized coefficients assigned to the same collection of "forced" variables. In concordance of predictions for individual patients in the generating sets, the overall proportions of disagreement for pairs of methods ranged from 0 to 28%, and kappa values ranged from 0.49 to 1.00. The accuracy of individual predictions showed relatively similar results when the different methods were applied to the same generating set. Across the generating sets, the different methods showed similar total results but substantial variations in predictions for alive and dead patients. When the models from the generating sets were applied for predictions in the challenge set, the results showed an analogous pattern: similar accuracy within models for overall and live/dead predictions, but substantial variations in live/dead predictions across models derived from different generating sources. The results showed that the multivariable methods often had good agreement with one another in predictions for groups but not for individual persons; and that no single method was superior to the others or to the composite staging system. We conclude that multivariable analytic methods may be most effective and consistent if used to find the few most powerful predictor variables, omitting the many other variables that may be "statistically significant" but less cogent. The powerful predictors may sometimes be best constructed, before the analysis begins, as composite variables containing appropriate unions or ordinal arrangements of elemental candidate variables.