SEARCH · Search PubMed

Results for “External validation”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3Linked to original sources

PubMed2005

Substantial effective sample sizes were required for external validation studies of predictive logistic regression models.

BACKGROUND AND OBJECTIVES: The performance of a prediction model is usually worse in external validation data compared to the development data. We aimed to determine at which effective sample sizes (i.e., number of events) relevant differences in model performance can be detected with adequate power. METHODS: We used a logistic regression model to predict the probability that residual masses of patients treated for metastatic testicular cancer contained only benign tissue. We performed standard power calculations and Monte Carlo simulations to estimate the numbers of events that are required to detect several types of model invalidity with 80% power at the 5% significance level. RESULTS: A validation sample with 111 events was required to detect that a model predicted too high probabilities, when predictions were on average 1.5 times too high on the odds scale. A decrease in discriminative ability of the model, indicated by a decrease in the c-statistic from 0.83 to 0.73, required 81 to 106 events, depending on the specific scenario. CONCLUSION: We suggest a minimum of 100 events and 100 nonevents for external validation samples. Specific hypotheses may, however, require substantially higher effective sample sizes to obtain adequate power.

Humans↗

PubMed2004

External validation and comparison of recently described prediction rules for suspected pulmonary embolism.

PURPOSE OF REVIEW: The assessment of pretest probability, allowing the categorization of patients clinically suspected of having pulmonary embolism in low, intermediate, and high clinical probability, is an essential step in contemporary diagnostic strategies because it permits limiting the number of additional diagnostic tests, especially invasive tests. Clinical probability can be evaluated implicitly or by prediction rules. Two prediction rules for pulmonary embolism have been described: the Canadian prediction rule (the Wells score) and the Geneva prediction rule. Their original descriptions were published in 2000 and 2001, respectively. These prediction rules need to be externally validated, and, ideally, outcome studies should demonstrate that patients may be safely treated on the basis of the assessment of the clinical probability they provide. Therefore, the purpose of this review is to discuss the external validation of these rules, because this particular point has been only recently achieved. RECENT FINDINGS: Application of both rules in an external setting and in a prospective study have confirmed their validity. A recent study suggests that the best evaluation is probably based on a prediction rule associated with possible clinical override. SUMMARY: Studies comparing an empiric assessment with explicit assessment, such as the Wells simplified score or the Geneva score, have shown that the three tools show similar accuracy. External validation and use of both rules in prospective management studies have only recently been performed and have confirmed their validity. Some reports suggest that empiric assessment may be influenced by level of training. Objective prediction rules seems to be less influenced by experience and should be preferred by more junior doctors. The tool used for clinical probability assessment is probably less important than the principle of a careful clinical probability assessment in each patient with suspected pulmonary embolism.

Diagnostic Techniques, Respiratory System↗

PubMed2006

External validation of a percutaneous coronary intervention mortality prediction model in patients with acute coronary syndromes.

BACKGROUND: The recently published Michigan outcome prediction model (MM) for inhospital mortality was developed and validated on a series of consecutive patients undergoing percutaneous coronary intervention (PCI). Our purpose was to externally validate the performance of the MM in 2 separate cohorts of patients with acute coronary syndrome (ACS) undergoing PCI in Canada. METHODS: A validation of the MM and development of an extended MM were performed on data describing 10,050 patients from the APPROACH prospective cohort study between January 1995 and December 2000. Performance of both models was assessed on an external data set of 3259 PCI cases from the British Columbia Cardiac Registries. Only patients with a diagnosis of ACS were included in the study. RESULTS: The original MM predicted death rates ranging from 0.1% to 60.6%, but lacked accuracy to predict inhospital mortality as severity increased. The extended MM predicted death rates more widely from 0.0% to a high of 91.0% with better accuracy to predict inhospital death in patients with ACS undergoing PCI. The areas under the receiver operating characteristic curve for the MM and the extended MM on the external validation data set were 0.93 and 0.95, respectively. CONCLUSION: The MM predicts death after PCI in patients with ACS and identifies a clear gradient of risk. However, the enhanced MM developed specifically for the subset of patients with ACS demonstrated better prediction and cross-validated performance. These prediction rules can be useful for risk-adjustment analyses and for prognostication for individual patients.

Aged↗

PubMed2003

External validity of a randomised clinical trial of temporomandibular disorders: analysis of the patients who refused to participate in research.

PURPOSE: To assess the external validity of a randomised clinical trial (RCT) of a painful condition. METHOD: Consecutive patients with painful temporomandibular disorders (TMDs) were invited to participate in a clinical trial. Patients who refused to participate were compared to those who agreed to participate in this study with respect to degree of symptoms at time of presentation. RESULTS: The patients who refused to participate had more pain, and their condition interfered more with their daily life than those who accepted the invitation to participate. CONCLUSION: Selection bias in RCTs of painful conditions can skew the results, and external validity should be analysed before the results are generalised.

Activities of Daily Living↗

PubMed2003

Prediction of outcome in acute lower-gastrointestinal haemorrhage based on an artificial neural network: internal and external validation of a predictive model.

BACKGROUND: Models based on artificial neural networks (ANN) are useful in predicting outcome of various disorders. There is currently no useful predictive model for risk assessment in acute lower-gastrointestinal haemorrhage. We investigated whether ANN models using information available during triage could predict clinical outcome in patients with this disorder. METHODS: ANN and multiple-logistic-regression (MLR) models were constructed from non-endoscopic data of patients admitted with acute lower-gastrointestinal haemorrhage. The performance of ANN in classifying patients into high-risk and low-risk groups was compared with that of another validated scoring system (BLEED), with the outcome variables recurrent bleeding, death, and therapeutic interventions for control of haemorrhage. The ANN models were trained with data from patients admitted to the primary institution during the first 12 months (n=120) and then internally validated with data from patients admitted to the same institution during the next 6 months (n=70). The ANN models were then externally validated and direct comparison made with MLR in patients admitted to an independent institution in another US state (n=142). FINDINGS: Clinical features were similar for training and validation groups. The predictive accuracy of ANN was significantly better than that of BLEED (predictive accuracy in internal validation group for death 87% vs 21%; for recurrent bleeding 89% vs 41%; and for intervention 96% vs 46%) and similar to MLR. During external validation, ANN performed well in predicting death (97%), recurrent bleeding (93%), and need for intervention (94%), and it was superior to MLR (70%, 73%, and 70%, respectively). INTERPRETATION: ANN can accurately predict the outcome for patients presenting with acute lower-gastrointestinal haemorrhage and may be generally useful for the risk stratification of these patients.

Acute Disease↗

PubMed1988

Improving depression severity assessment--II. Content, concurrent and external validity of three observer depression scales.

The Hamilton Depression Scale (HAMD), the Montgomery-Asberg Depression Rating Scale (MADRS) and the Bech-Rafaelsen Melancholia Scale (BRMS) were compared with respect to content, concurrent and external validity in sample of 130 patients with a major depressive episode. The three scales did equally well in concurrent and external validity. The HAMD showed some deficiencies in content validity. The consequences for depression severity assessment are discussed.

Adult↗

PubMed2005

Factor structure and external validity of the PANSS revisited.

Considerable controversy exists concerning the positive and negative syndrome scale (PANSS), one of the most widely used instruments in schizophrenia research. In this article we revisited the factor structure and external validity of the PANSS in a sample of 500 participants with DSM IV diagnoses of schizophrenia. We found that a model with six latent factors provided a relatively good fit, considered adequate by two rules of thumb. Five factors corresponded closely to those typically derived in other studies: Negative, Positive, Excited/Activation, Anxious-Depressed/Dysphoric, and Disorganized/Autistic preoccupation. The sixth factor seemed to have face validity and was labeled Withdrawn. With the exception of Anxious-Depressed/Dysphoric, Cronbach's Alpha ranged from 0.70 to 0.85 suggesting an acceptable internal consistency. External validity was studied through correlations with socio-demographic variables, DSM IV (subtype) diagnoses, clinical characteristics, and drug use. The many significant correlations suggested that the six PANSS scales measure meaningful aspects of schizophrenia. Furthermore, the pattern of correlations varied, providing evidence that the scales assessed partly different aspects of the disease. Our analyses also suggested that some of the controversy about the PANSS can possibly be attributed to methodological factors where the substantial cross-loadings of some PANSS items may play an important role.

Adult↗

PubMed2000

The effect of knee angle on the external validity of isometric measures of lower body neuromuscular function.

The aim of this study was to evaluate the effect of varying knee angle (120 degrees and 90 degrees) on the external validity of an isometric leg press test with reference to vertical jump performance. Isometric peak force (PF120 and PF90), rate of force development (RFD120 and RFD90), and maximum height reached with a squat jump and counter-movement jump were measured in 14 males. Although RFD120 was significantly correlated with squat jump and counter-movement jump performance (r = 0.71 and 0.69), and the correlations with PF120 approached statistical significance (r = 0.53 and 0.50), neither PF90 nor RFD90 was significantly related to vertical jump performance. Furthermore, although both RFD120 and PF120 were significantly different between the best five and the worst five jumpers, RFD90 and PF90 did not differentiate between individuals' vertical jump performance. We conclude that the choice of joint angle affects the external validity of isometric strength testing. Based on our results, we recommend accurate control of biomechanical specificity and assessment at different angles to find the position at which isometric strength testing is most comfortable.

Adult↗

PubMed1992

Panel attrition and external validity in adolescent substance use research.

Panel attrition threatens external validity in adolescent substance use research. A 7-year adolescent panel was examined to determine whether attrition effects varied by (a) type of substance assessed and (b) method of measurement and type of statistical analysis. Chi-squares and multivariate analyses of variance revealed that study dropouts were more likely to use substances and reported higher mean use of substances at baseline than stayers; attrition effects varied by substance; and mean use comparisons were more likely to detect attrition effects than use-nonuse comparisons. Implications of these findings for adolescent substance use research are discussed.

Adolescent↗

PubMedDate not supplied

Research methodology for the Internet: external validity (generalizability).

The burgeoning use of the Internet for nursing research calls for a need to examine characteristics of electronic populations, and how they can be conceptualized relative to their physically based counterparts. The 1963 work of Campbell and Stanley, now 40 years old, is revisited in relation to external validity, and its applicability to today's world of quantitative and qualitative research. Internal and external differences in Internet populations are outlined, and a new threat to external validity is proposed: the threat of Networked Effects, which is composed of 4 subfactors: Co-occurring Interferences to Testing, Self-selection Mechanisms, Electronic Group Environments, and Cultural Differences. These Internet threats must be well understood when formulating research methodology.

Humans↗

PubMed2003

Age and National Institutes of Health Stroke Scale Score within 6 hours after onset are accurate predictors of outcome after cerebral ischemia: development and external validation of prognostic models.

BACKGROUND AND PURPOSE: To date, no validated, comprehensive, and practicable model exists to predict functional recovery within the first hours of cerebral ischemic symptoms. The purpose of this study was to externally validate 2 prognostic models predicting functional outcome and survival at 100 days within the first 6 hours after onset of acute cerebral ischemia. METHODS: On admission to a participating hospital, patients were registered prospectively and included according to defined criteria. Follow-up was performed 100 days after the event. With the use of prospectively collected data, 2 prognostic models were developed and internally calibrated in 1079 patients and externally validated in 1307 patients. By means of age and National Institutes of Health Stroke Scale (NIHSS) score as independent variables, model I predicts incomplete functional recovery (Barthel Index <95) versus complete functional recovery, and model II predicts mortality versus survival. RESULTS: In the validation data set, model I correctly predicted 62.9% of the patients who were incompletely restituted or had died and 83.2% of the completely restituted patients, and model II correctly predicted 57.9% of the patients who had died and 91.5% of the surviving patients. Both models performed better than the treating physicians' predictions made within 6 hours after admission. CONCLUSIONS: The resulting prognostic models are useful to correctly stratify treatment groups in clinical trials and should guide inclusion criteria in clinical trials, which in turn increases the power to detect clinically relevant differences.

Age Factors↗

PubMed2006

Epidural analgesia associated with low-dose oxytocin augmentation increases cesarean births: a critical look at the external validity of randomized trials.

OBJECTIVE: Randomized controlled trials suggest epidural analgesia (EA) does not increase the frequency of cesarean births compared with opioid analgesia. We analyzed trials comparing EA with opioid analgesia to determine their external validity in contemporary North American practice. STUDY DESIGN: Randomized controlled trials comparing EA with opioid analgesia were identified from the Cochrane database and Medline and included if they reported labor outcomes and management protocols. Labor management was then compared with current obstetric practice determined from surveys of North American teaching maternity units and clinical practice guidelines. RESULTS: Of 19 trials identified, 8 were included. Seven trials used Active Management of Labor protocols that used high-dose oxytocin; each demonstrated no epidural-related increase in cesarean births. One trial that used low-dose oxytocin demonstrated a marked increase in cesarean births. Most large North American obstetric units use low-dose oxytocin. CONCLUSION: Randomized trials showing no effect of EA on cesarean section (CS) rate lack external validity in much of North American practice. The limited data available suggest EA and low-dose oxytocin used together increase the CS rate. Early detection of dystocia and high-dose oxytocin augmentation should be considered for women receiving EA; those delivering in low-dose oxytocin settings should be advised of a probable increase in the likelihood of CS.

Analgesia, Epidural↗

PubMed1999

External validation of IASP diagnostic criteria for Complex Regional Pain Syndrome and proposed research diagnostic criteria. International Association for the Study of Pain.

Recent work in our research consortium has raised internal validity concerns regarding the current IASP criteria for Complex Regional Pain Syndrome (CRPS), suggesting problems with inadequate sensitivity and specificity. The current study explored the external validity of these IASP criteria for CRPS. A standardized evaluation of signs and symptoms of CRPS was conducted by study physicians in 117 patients meeting IASP criteria for CRPS, and 43 patients experiencing neuropathic pain with established non-CRPS etiology (e.g. diabetic neuropathy, post-herpetic neuralgia). Multiple discriminant function analyses were used to test the ability of the IASP diagnostic criteria and decision rules, as well as proposed research modifications of these criteria, to discriminate between CRPS patients and those experiencing non-CRPS neuropathic pain. Current IASP criteria and decision rules (e.g. signs or symptoms of edema, or color changes or sweating changes satisfy criterion 3) discriminated significantly between groups (P < 0.001). However, although sensitivity was quite high (0.98), specificity was poor (0.36), and a positive diagnosis of CRPS was likely to be correct in as few as 40% of cases. Empirically-based research modifications to the criteria, which are more comprehensive and require presence of signs and symptoms, were also tested. These modified criteria were also able to discriminate significantly, between the CRPS and non-CRPS groups (P < 0.001). A decision rule, requiring at least two sign categories and four symptom categories to be positive optimized diagnostic efficiency, with a diagnosis of CRPS likely to be accurate in up to 84% of cases, and a diagnosis of non-CRPS neuropathic pain likely to be accurate in up to 88% of cases. These results indicate that the current IASP criteria for CRPS have inadequate specificity and are likely to lead to overdiagnosis. Proposed modifications to these criteria substantially improve their external validity and merit further evaluation.

Adult↗

PubMed2003

External validation of a housing recommendation model for clients following stroke rehabilitation.

PURPOSE: Using a Social Judgment Theory approach, mathematical models of clinical decisions can be developed to predict new decisions. However, since these models are often developed under 'laboratory conditions' using hypothetical cases rather than real world client samples, questions may be raised concerning the external validity of such models. The purpose of this research was to validate a discharge housing decision making model developed with hypothetical clients with stroke with a real sample. This process is referred to as external validation. METHODS: Cross-tabulations were performed to compare the model discharge housing prediction against the team's actual recommendations for 60 clients from three rehabilitation centres. A sensitivity and specificity analysis was used to compare the model predictions against the client's actual discharge housing. RESULTS: A simple mathematical model was constructed to accurately predict housing recommendations. However, the predictive power of the model appeared to be limited by clinician/team consideration of unmeasured factors such as client personality and motivation. CONCLUSION: It is suggested that the housing decision model developed may be useful for rehabilitation teams as a starting point when discussing the best place to recommend a client reside on discharge from rehabilitation. Such a model may assist teams to standardise their decision processes and minimise the potential for unwarranted bias.

Decision Support Techniques↗

PubMed2006

External validation of outcome prediction model for ureteral/renal calculi.

PURPOSE: We externally validated a previously designed neural network model to predict outcome and duration of passage for ureteral/renal calculi. The model was also evaluated using a 6 mm largest stone dimension cutoff in predicting stone outcome. MATERIALS AND METHODS: The model was previously designed on 301 patients at Albany Medical Center (free shareware from www.uroengineering.com). The model had a prediction accuracy of 86% for passage outcome and 87% for passage duration. In this study we tested the model on a separate 384 patients from 6 different external institutions to assess the prediction accuracy. All patients had a single renal/ureteral calculus by evaluation in an emergency room setting or by primary physicians and were then referred for further treatment. Model accuracy was also compared to using a 6 mm largest stone dimension cutoff in predicting the need for intervention. RESULTS: Testing on the 384 patients from all 6 external institutions revealed an outcome prediction accuracy of 88%. The area under the ROC curve was 0.9. Using a 6 mm stone size cutoff provided 79% (ROC 0.8) accuracy. The model duration of passage prediction accuracy was 80% (133 patients passed the stone, area under ROC of 0.8). CONCLUSIONS: The model provided high stone outcome prediction accuracy (ROC of 0.9 and 0.8) at the 6 external institutions, comparable to that of the design institution. The model provided higher accuracy than using only the largest stone dimension as a cutoff. Increasing experience will further assess the model's accuracy.

Adolescent↗

PubMed2026

Development and external validation of an explainable machine learning model for predicting chronic kidney disease progression in the Korean population.

BACKGROUND: Current risk stratification models, such as the Kidney Failure Risk Equation (KFRE), exhibit variable performance across ethnic groups and fail to capture dynamic clinical trajectories. This study aimed to develop and validate a Korean-specific machine learning (ML) model for predicting chronic kidney disease (CKD) progression using an ensemble approach. METHODS: We used electronic health records from Seoul National University Hospital for model development (n = 28,209) and the Korean Genome and Epidemiology Study (KoGES) CKD cohort for external validation (n = 3,960). The primary outcome was a composite of ≥40% decline in estimated glomerular filtration rate (eGFR) or progression to end-stage renal disease within 2 years. A soft-voting ensemble of four ML algorithms (XGBoost, LightGBM, CatBoost, and Random Forest) was developed. RESULTS: The ensemble model demonstrated robust discrimination in internal validation (area under the receiver operating characteristic curve [AUROC], 0.939; 95% confidence interval [CI], 0.934-0.944), significantly exceeding the KFRE (AUROC, 0.879-0.884). External validation in the KoGES cohort showed comparable discrimination (AUROC, 0.859; 95% CI, 0.798-0.914) versus KFRE (four-variable AUROC, 0.882; 95% CI, 0.818-0.935). Shapley Additive exPlanations (SHAP) analysis identified baseline eGFR, serum creatinine, eGFR slope, albumin, and hemoglobin as key prognostic features, supporting a complementary framework using KFRE for community screening and the ML model for hospital-based risk stratification. CONCLUSION: The ensemble ML model accurately predicts short-term CKD progression in Korean patients. By incorporating longitudinal features and ensemble learning, it provides a precise alternative to Western-derived equations, particularly in tertiary care settings.

Chronic kidney failure↗

PubMed2003

Predicting outcome in ischemic stroke: external validation of predictive risk models.

BACKGROUND: Six multivariable models predicting 3-month outcome of acute ischemic stroke have been developed and internally validated previously. The purpose of this study was to externally validate the previous models in an independent data set. SUMMARY OF REPORT: We predicted outcomes for 299 patients with ischemic stroke who received placebo in the National Institute of Neurological Disorders and Stroke rt-PA trial. The model equations used 6 acute clinical variables and head CT infarct volume at 1 week as independent variables and 3-month National Institutes of Health Stroke Scale, Barthel Index, and Glasgow Outcome Scale as dependent variables. Previously developed model equations were used to forecast excellent and devastating outcome for subjects in the placebo tissue plasminogen activator data set. Area under the receiver operator characteristic curve was used to measure discrimination, and calibration charts were used to measure calibration. The validation data set patients were more severely ill (National Institutes of Health Stroke Scale and infarct volume) than the model development subjects. Area under the receiver operator characteristic curves demonstrated remarkably little degradation in the validation data set and ranged from 0.75 to 0.89. Calibration curves showed fair to good calibration. CONCLUSIONS: Our models have demonstrated excellent discrimination and acceptable calibration in an external data set. Development and validation of improved models using variables that are all available acutely are necessary.

Aged↗

PubMed1987

Do smoking prevention programs really work? Attrition and the internal and external validity of an evaluation of a refusal skills training program.

This study investigated the effects of a smoking prevention program that emphasized refusal skills training on 1730 adolescents in three high schools and six middle schools. Classes within these schools were randomly assigned to treatment or no-treatment conditions to avoid confounding schools with treatment condition. The effects of attrition on the internal and external validity of the study were examined. Although the results indicated an apparent effect of the program at the 1-year follow-up in deterring continued smoking among those who were smoking at pretest, this result may have been due to a higher rate of attrition among high-rate smokers in the treatment condition than in the control condition. Attrition also affected external validity. Across both conditions, subjects who were smoking at pretest and who were at risk to smoke were more likely to be missing at follow-up. The program did have an effect on the refusal skills of participants and the validity of this effect was not jeopardized by differential attrition.

Adolescent↗