PubMed1999
OBJECTIVE: To conduct a multicenter study to validate the accuracy of the Acute Physiology and Chronic Health Evaluation (APACHE) II system, APACHE III system, Trauma and Injury Severity Score (TRISS) methodology, and a 24-hour intensive care unit (ICU) point system for prediction of mortality in ICU trauma patient admissions. METHODS: The study population consisted of retrospectively identified, consecutive ICU trauma admissions (n = 2,414) from six Level I trauma centers. Probabilities of death were calculated by using logistic regression analysis. The predictive power of each system was evaluated by using decision matrix analysis to compare observed and predicted outcomes with a decision criterion of 0.50 for risk of hospital death. The Youden Index (YI) was used to compare the proportion of patients correctly classified by each system. Measures of model calibration were based on goodness-of-fit testing (Hosmer-Lemeshow statistic less than 15.5) and model discrimination were based on the area under the receiver operating characteristic curve (AUC). RESULTS: Overall, APACHE II (sensitivity, 38%; specificity, 99%; YI, 37%; H-L statistic, 92.6; AUC, 0.87) and TRISS (sensitivity, 52%; specificity, 94%; YI, 46%; H-L statistic, 228.1; AUC, 0.82) were poor predictors of aggregate mortality, because they did not meet the acceptable thresholds for both model calibration and discrimination. APACHE III (sensitivity, 60%; specificity, 98%; YI, 58%; H-L statistic, 7.0; AUC, 0.89) was comparable to the 24-hour ICU point system (sensitivity, 51%; specificity, 98%; YI, 50%; H-L statistic, 14.7; AUC, 0.89) with both systems showing strong agreement between the observed and predicted outcomes based on acceptable thresholds for both model calibration and discrimination. The APACHE III system significantly improved upon APACHE II for estimating risk of death in ICU trauma patients (p < 0.001). Compared with the overall performance, for the subset of patients with nonoperative head trauma, the percentage correctly classified was decreased to 46% for APACHE II; increased to 71% for APACHE III (p < 0.001 vs. APACHE II); increased to 59% for TRISS; and increased to 62% for 24-hour ICU points. For operative head trauma, the percentage correctly classified was increased to 60% for APACHE II; increased to 61% for APACHE III; decreased to 43% for TRISS (p < 0.004 vs. APACHE III); and increased to 54% for 24-hour ICU points. For patients without head injuries, all of the systems were unreliable and considerably underestimated the risk of death. The percentage of nonoperative and operative patients without head trauma who were correctly classified was decreased, respectively, to 26% and 30% for APACHE II; 33% and 29% for APACHE III; 33% and 19% for TRISS; 20% and 23% for 24-hour ICU points. CONCLUSION: For the overall estimation of aggregate ICU mortality, the APACHE III system was the most reliable; however, performance was most accurate for subsets of patients with head trauma. The 24-hour ICU point system also demonstrated acceptable overall performance with improved performance for patients with head trauma. Overall, APACHE II and TRISS did not meet acceptable thresholds of performance. When estimating ICU mortality for subsets of patients without head trauma, none of these systems had an acceptable level of performance. Further multicenter studies aimed at developing better outcome prediction models for patients without head injuries are warranted, which would allow trauma care providers to set uniform standards for judging institutional performance.