SEARCH · Search PubMed

Results for “External validation”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources

PubMed2002

Studying pretrial publicity effects: new methods for improving ecological validity and testing external validity.

Although research examining the effects of pretrial publicity (PTP) on individuals' appraisals of a defendant and verdict decision making generally has been found to be internally valid, the external validity has been questioned by some social scientists as well as lawyers and judges. It is often proposed that the verisimilitude (or ecological validity) ofthe research should be increased in the service of increasing external validity; however, increasing verisimilitude can be costly in terms of both time and money. It is proposed that the Internet is a viable means of conducting PTP research that allows high verisimilitude without high costs. This is demonstrated with a study in which we used the Internet to examine PTP effects in an actual trial as it was taking place. Successful use of the Internet to conduct experimental research in other areas of psychology and law is discussed, as well as the importance of future research examining whether independent variables interact with methods in ways that undermine the generalizability of research findings.

Adult↗

PubMed2006

External validity: the neglected dimension in evidence ranking.

Evidence that is both accurate (internally valid) and relevant (externally valid) is needed to decide which treatment is best for a particular patient. Evidence rankings facilitate the marshalling of evidence on clinical decisions in the common context of an overwhelming number of studies, some with conflicting results. Evidence from randomized control trials is typically ranked above evidence from non-experimental studies since rankings are based primarily, if not exclusively, on considerations of internal validity. We propose that evidence rankings should consider equally both internal and external validity. External validity includes how closely the study population, the institution types in the study, the types of physicians in the study, the role of clinician decision-making (e.g. dose adjustment) in the study, and the role of patient preferences in the study resemble those in actual practice. The example of spironolactone use in heart failure illustrates the danger in using evidence that is internally but not externally valid. Ideally, a treatment should only be used when both internally and externally valid evidence indicates that it will be useful for the particular patient.

Case-Control Studies↗

PubMed2003

Internal and external validation of predictive models: a simulation study of bias and precision in small samples.

We performed a simulation study to investigate the accuracy of bootstrap estimates of optimism (internal validation) and the precision of performance estimates in independent validation samples (external validation). We combined two data sets containing children presenting with fever without source (n=376+179=555; 120 bacterial infections). Random samples were drawn from this combined data set for the development (n=376) and validation (n=179) of logistic regression models. The models included statistically significant predictors for infection selected from a set of 57 candidate predictors. Model development, including the selection of predictors, and validation were repeated in a bootstrapping procedure. The resulting expected optimism estimate in the receiver operating characteristic (ROC) area was compared with the observed optimism according to independent validation samples. The average apparent ROC area was 0.74, which was expected (based on bootstrapping) to decrease by 0.07 to 0.67, whereas the observed decrease in the validation samples was 0.09 to 0.65. Omitting the selection of predictors from the bootstrap procedure led to a severe underestimation of the optimism (decrease 0.006). The standard error of the observed ROC area in the independent validation samples was large (0.05). We recommend bootstrapping for internal validation because it gives reasonably valid estimates of the expected optimism in predictive performance provided that any selection of predictors is taken into account. For external validation, substantial sample sizes should be used for sufficient power to detect clinically important changes in performance as compared with the internally validated estimate.

Bias↗

PubMed1989

Internal and external validity in two studies that compared treatment methods.

Research comparing the effectiveness of two treatments offers both strengths and weaknesses for occupational therapy. Although it is worthwhile to determine which of two treatments works best for a particular problem, methodological problems may arise that preclude a valid conclusion. To draw valid conclusions from research, criteria for internal validity and external validity must be satisfied. The two preceding articles in this issue are examples of studies that used between-groups experimental methodology to compare the effectiveness of two different treatments. This paper evaluates the above-mentioned studies on the basis of principles of internal and external validity. One of these studies (Jongbloed, Stacey, & Brighton, 1989) was truly experimental, whereas the other study (Groves & Rider, 1989) was quasi-experimental. Results from both studies were similar because, in each study, both of the treatment groups improved, but there were no significant differences between treatments. Absence of a true control group in both studies presented limitations on the conclusion that both treatments worked equally well.

Analysis of Variance↗

PubMed2006

Computer-assisted detection for CT colonography: external validation.

AIM: To externally validate a computer-assisted detection (CAD) system for computed tomography (CT) colonography, using data from a single centre uninvolved with the software development. MATERIALS AND METHODS: Twenty-five multi-detector CT colonography examinations of patients with validated polyps accumulated at a single centre were examined by two readers who used endoscopic and histopathological data to identify polyp coordinates. A CAD system that had been developed using data from elsewhere, and had not previously encountered the present data, was then applied to the data at sphericity filter settings of 0.75 and 0.50 and identified potential polyps. True-positive, false-negative, and false-positive counts were determined by comparison with the known polyp coordinates. RESULTS: Twenty-five patients had 57 polyps, median size 6mm (range 1-15mm). Per-patient sensitivity for the CAD system was 96% (24 of 25). The CAD system detected 44 (77%) polyps at sphericity setting 0.75 and 49 (86%) polyps at sphericity 0.50: the additional five polyps detected all measured 5mm or less. Sphericity of 0.75 resulted in a median of 10 (one to 34) easily dismissed false-positive prompts per patient and a median of 4 (zero to 15) that needed three-dimensional rendering before dismissal. This rose to 32 (16 to 99) and 11 (three to 35), respectively, at sphericity 0.5. CONCLUSIONS: A per-patient sensitivity of 96% was found for the CAD system (in patients with a median polyp diameter of 6mm) using external validation, a more stringent test than either internal cross-validation or temporal validation. Decreasing sphericity increases sensitivity for small polyps at the expense of decreased specificity.

Colonic Polyps↗

PubMed2007

Development and external validation of an extended repeat biopsy nomogram.

PURPOSE: We hypothesized that the outcome of repeat biopsy could be accurately predicted. We tested this hypothesis in a contemporary cohort from 3 centers. MATERIALS AND METHODS: The principal cohort of 1,082 men from Hamburg, Germany was used for nomogram development as well as for internal 200 bootstrap validation in 721 and external validation in 361. Two additional external validation cohorts, including 87 men from Milan, Italy and 142 from Seattle, Washington, were also used. Predictors of prostate cancer on repeat biopsy were patient age, digital rectal examination, prostate specific antigen, percent free prostate specific antigen, number of previous negative biopsy sessions and sampling density. Multivariate logistic regression models were used to develop the nomograms. RESULTS: The mean number of previous negative biopsies was 1.5 (range 1 to 6) and the mean number of cores at final repeat biopsy was 11.1 (range 10 to 24). Of the men 370 (30.2%) had prostate cancer. On multivariate analyses all predictors were statistically significant (p < or =0.028). After internal validation the nomogram was 76% accurate. External validation showed 74% (Hamburg), 78% (Milan) and 68% (Seattle) accuracy. CONCLUSIONS: Relative to the previous nomograms (10 predictors or 71% accuracy) our tool relies on fewer variables (6) and shows superior accuracy in European men. Accuracy in American men is substantially lower. Racial, clinical and biochemical differences may explain the observed discrepancy in predictive accuracy.

Adult↗

PubMed1983

Family planning field research projects: balancing internal against external validity.

This report discusses the experience of a two-year family planning and maternal/child health project in Nepal. Although the project was planned as an experimental field research endeavor, a series of unanticipated events repeatedly compromised the internal validity of the project and forced design changes. While unexpected events are common in the history of most field projects, they present the research evaluator with the fundamental dilemma of trying to maintain a high degree of internal validity without sacrificing external validity. Rigid research designs with tight control over the introduction and measurement of experimental variables may serve to increase internal validity but they may also create an atypical and artificial situation that fails to mirror real field conditions and thus threatens external validity.

Community Health Services↗

PubMed2004

External validity, generalizability, and knowledge utilization.

PURPOSE: To examine the concepts of external validity and generalizability, and explore strategies to strengthen generalizability of research findings, because of increasing demands for knowledge utilization in an evidence-based practice environment. FRAMEWORK: The concepts of external validity and generalizability are examined, considering theoretical aspects of external validity and conflicting demands for internal validity in research designs. Methodological approaches for controlling threats to external validity and strategies to enhance external validity and generalizability of findings are discussed. CONCLUSIONS: Generalizability of findings is not assured even if internal validity of a research study is addressed effectively through design. Strict controls to ensure internal validity can compromise generalizability. Researchers can and should use a variety of strategies to address issues of external validity and enhance generalizability of findings. Enhanced external validity and assessment of generalizability of findings can facilitate more appropriate use of research findings.

Data Collection↗

PubMed2006

Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity.

BACKGROUND: It is often stated that external validity is not sufficiently considered in the assessment of clinical studies. Although tools for its evaluation have been established, there is a lack of awareness of their significance and application. In this article, a comprehensive checklist is presented addressing these relevant criteria. METHODS: The checklist was developed by listing the most commonly used assessment criteria for clinical studies. Additionally, specific lists for individual applications were included. The categories of biases of internal validity (selection, performance, attrition and detection bias) correspond to structural, treatment-related and observational differences between the test and control groups. Analogously, we have extended these categories to address external validity and model validity, regarding similarity between the study population/conditions and the general population/conditions related to structure, treatment and observation. RESULTS: A checklist is presented, in which the evaluation criteria concerning external validity and model validity are systemized and transformed into a questionnaire format. CONCLUSION: The checklist presented in this article can be applied to both planning and evaluating of clinical studies. We encourage the prospective user to modify the checklists according to the respective application and research question. The higher expenditure needed for the evaluation of clinical studies in systematic reviews is justified, particularly in the light of the influential nature of their conclusions on therapeutic decisions and the creation of clinical guidelines.

Bias↗

PubMed2002

Internal and external validation of the NOSEP prediction score for nosocomial sepsis in neonates.

OBJECTIVE: To evaluate the performance of a scoring system (NOSEP) to predict nosocomial sepsis in neonates at the hospital where the score was developed (internal validation) and in an independent data set from other centers (external validation). DESIGN: Multiple center prospective cohort study. SETTING: Six neonatal intensive care units from the Flanders in Belgium. PATIENTS: We analyzed two groups of patients: 62 episodes of presumed nosocomial sepsis in the internal validation cohort and 93 episodes of presumed nosocomial sepsis in a multiple center external validation cohort. INTERVENTIONS: Assessment of the predictive power of the NOSEP score 24 hrs preceding sepsis workup and the patients' basic demographic characteristics and co-morbidity was performed. Diagnosis of nosocomial sepsis and the microbiology results were registered. MAIN RESULTS: The NOSEP score's discriminative capability was very good in the internal validation (area under receiver operating characteristic curve = 0.73 +/- 0.08 [sem]). The NOSEP score performed satisfactory in the external validation (area under receiver operating characteristic curve = 0.66 +/- 0.06). The calibration capability in both validation sets as measured by goodness-of-fit tests (internal validation, p =.56; external validation, p =.48) was good. An improvement of the NOSEP score was obtained for the external centers by redefining the cut-off of the items of the NOSEP score (area under receiver operating characteristic curve for NOSEP-NEW-I = 0.71 +/- 0.05) or adding co-morbidity factors (area under receiver operating characteristic curve for NOSEP-NEW-II = 0.82 +/- 0.04), with good calibration performance (goodness-of-fit test, p >.50). Finally, the fit of the NOSEP score demonstrated no significant variation across subgroups of patients. CONCLUSIONS: The predictive power of the original NOSEP score is very good in neonates at the original neonatal intensive care unit. In other neonatal intensive care units, its discriminatory performance is satisfactory but could be improved after modification of the variables in the model or adding additional variables. To use such a NOSEP score in other neonatal intensive care units, its accuracy has to be validated and adjusted if necessary.

Cross Infection↗

PubMed2003

External validation is necessary in prediction research: a clinical example.

BACKGROUND AND OBJECTIVES: Prediction models tend to perform better on data on which the model was constructed than on new data. This difference in performance is an indication of the optimism in the apparent performance in the derivation set. For internal model validation, bootstrapping methods are recommended to provide bias-corrected estimates of model performance. Results are often accepted without sufficient regard to the importance of external validation. This report illustrates the limitations of internal validation to determine generalizability of a diagnostic prediction model to future settings. METHODS: A prediction model for the presence of serious bacterial infections in children with fever without source was derived and validated internally using bootstrap resampling techniques. Subsequently, the model was validated externally. RESULTS: In the derivation set (n=376), nine predictors were identified. The apparent area under the receiver operating characteristic curve (95% confidence interval) of the model was 0.83 (0.78-0.87) and 0.76 (0.67-0.85) after bootstrap correction. In the validation set (n=179) the performance was 0.57 (0.47-0.67). CONCLUSION: For relatively small data sets, internal validation of prediction models by bootstrap techniques may not be sufficient and indicative for the model's performance in future patients. External validation is essential before implementing prediction models in clinical practice.

Bacterial Infections↗

PubMed2026

Externally validated risk prediction models for gestational diabetes mellitus: A systematic review and meta-analysis.

INTRODUCTION: Risk prediction models for gestational diabetes mellitus (GDM) offer potential for early identification and targeted prevention. External validation is crucial to assess model performance across diverse populations. Despite the availability of numerous GDM prediction models, limited evidence exists on their external validation frequency, methodological quality, and clinical applicability. This systematic review evaluated externally validated GDM prediction models, focusing on methodological rigor, reporting standards, and clinical relevance to inform future research and implementation. MATERIAL AND METHODS: Databases including Ovid MEDLINE, Embase, Scopus, Emcare, and CINAHL were searched up to May 1, 2025. Studies reporting external validation of GDM risk prediction models were included. Two reviewers independently screened studies. Data were extracted using the CHARMS framework, and risk of bias and applicability were assessed using PROBAST+AI. The study protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO; CRD420251125758). RESULTS: Twenty-six studies validated 33 models, with validation sample sizes ranging from 50 to 75 161. Over half used the IADPSG criteria to define GDM. Discrimination metrics were commonly reported, but calibration, overall performance, and clinical utility were often lacking. Meta-analysis was feasible for only four models: Teede et al., Nanda et al., Naylor et al., and Van Leeuwen et al., each showing fair discrimination. The Teede et al. model was the most widely validated, with 11 external validations across six continents and a pooled AUC of 0.72 (95% CI: 0.67-0.76). Despite fewer validations, the Nanda et al. model achieved the highest pooled discrimination (5 validations; pooled AUC 0.77, 95% CI: 0.74-0.80). The Naylor et al. and van Leeuwen et al. models also underwent meta-analysis, as sufficient external validation studies were available to support comparative performance assessment. Notably, 69.23% of studies had a high risk of bias. CONCLUSIONS: While many models showed acceptable predictive performance, most validations were methodologically weak. Future studies should follow best-practice guidelines and promote scalable validation strategies, such as algorithm sharing, to enhance clinical utility.

Humans↗

PubMed2001

Distinguishing the benign and malignant adnexal mass: an external validation of prognostic models.

OBJECTIVE: Because external validation of the present models has not been reported, the purpose of the present study was to assess existing diagnostic models that are used to distinguish malignant from benign masses. METHODS: We tested the performance of existing models in a prospectively assembled data set of 170 patients with an adnexal mass. Twenty-one models that have been reported previously were assessed. The models were based on combinations of ultrasound findings, color Doppler tests, CA-125 measurement, age, and/or menopausal status. For each model, we constructed ROC curves and calculated an area under the ROC curve. RESULTS: Of the 170 adnexal masses that were operated on, 30 (18%) were malignant. The area under the ROC curve of 21 models that were externally validated varied between 0.69 and 0.90. We found the performance of the existing models to be inferior to the performance reported in the initial studies. Even models that incorporated multiple diagnostic tools and that were developed using logistic regression models or neural networks had an area under the ROC curve of 0.86 at maximum. In the case where we focused on almost perfect sensitivity, the highest specificities varied between 0.45 and 0.60. CONCLUSION: Although diagnostic models might be of value in the preoperative assessment of the adnexal mass, their diagnostic performance is not as good as that reported in the original publications.

Adnexal Diseases↗

PubMed2006

Statistical external validation and consensus modeling: a QSPR case study for Koc prediction.

The soil sorption partition coefficient (log K(oc)) of a heterogeneous set of 643 organic non-ionic compounds, with a range of more than 6 log units, is predicted by a statistically validated QSAR modeling approach. The applied multiple linear regression (ordinary least squares, OLS) is based on a variety of theoretical molecular descriptors selected by the genetic algorithms-variable subset selection (GA-VSS) procedure. The models were validated for predictivity by different internal and external validation approaches. For external validation we applied self organizing maps (SOM) to split the original data set: the best four-dimensional model, developed on a reduced training set of 93 chemicals, has a predictivity of 78% when applied on 550 validation chemicals (prediction set). The selected molecular descriptors, which could be interpreted through their mechanistic meaning, were compared with the more common physico-chemical descriptors log K(ow) and log S(w). The chemical applicability domain of each model was verified by the leverage approach in order to propose only reliable data. The best predicted data were obtained by consensus modeling from 10 different models in the genetic algorithm model population.

Algorithms↗

PubMed2026

Clinical Variable-Based Machine Learning for Predicting Early mCRPC Using Exclusively Clinical Variables: Development and Multicenter External Validation.

BACKGROUND AND OBJECTIVE: Metastatic hormone-sensitive prostate cancer (mHSPC) exhibits heterogeneous progression patterns, with early progression to metastatic castration-resistant prostate cancer (mCRPC) within 12 months indicating aggressive tumor biology and poor prognosis. Current risk stratification tools (CHAARTED, LATITUDE) offer limited individualized prediction. Machine learning approaches are increasingly applied to predict prostate cancer progression, but most models show modest performance (AUC 0.68-0.72), limited external validation, or require genomic variables unavailable in routine practice. This study aimed to develop and externally validate a novel RINH algorithm for predicting early mCRPC progression (≤ 12 months) using exclusively clinical variables, positioning it as a superior alternative to conventional ML classifiers. METHODS: This multicenter study enrolled 412 patients with de novo mHSPC from seven Spanish academic centers using mixed retrospective-prospective data collection. Twenty clinical variables were recorded, including demographics, PSA, ISUP grade, metastatic localization, CHAARTED/LATITUDE classifications, and treatment modalities. Following RINH-based outlier exclusion (55 patients), 357 patients (29 with early progression, 8.1%) were used to train six ML algorithms: RINH, Logistic Regression, Linear Discriminant, Support Vector Machine, Random Forest, and Subspace Discriminant. A two-tiered validation strategy integrated stratified fivefold cross-validation across all centers and formal external validation using center 1 (n = 121, 19 events) for training and centers 2-7 (n = 207, 10 events) for independent testing. Performance metrics included AUC, sensitivity, specificity, accuracy, and F1-score. KEY FINDINGS AND LIMITATIONS: Artificial intelligence and machine learning (ML) are transforming oncology, promising personalized risk stratification beyond traditional clinical criteria. In metastatic hormone-sensitive prostate cancer (mHSPC), early progression to castration resistance (mCRPC) within 12 months signals aggressive biology and poor prognosis, yet current tools (CHAARTED, LATITUDE) offer limited individualized prediction. Multiple ML models have been proposed with variable success: most achieve modest performance (AUC 0.68-0.72), lack robust external validation, or rely on genomic variables inaccessible in routine practice. We propose a novel approach using the Rivality Index Neighborhood (RINH) algorithm, demonstrating superior predictive capacity in an initial multicenter validation with exclusively clinical variables. This study provides rigorous multicenter external validation, advancing toward implementable precision oncology tools. CONCLUSIONS AND CLINICAL IMPLICATIONS: The RINH algorithm achieves superior predictive performance for early mCRPC progression using exclusively clinical variables, representing a significant advance toward implementable risk stratification. However, low reliability scores in external validation underscore that excellent performance metrics alone do not guarantee stability. Before clinical deployment, validation in substantially larger cohorts with higher progression events is essential. If validated, this model could enable personalized, risk-adapted therapeutic strategies, refining patient selection for treatment intensification or de-escalation.

Humans↗

PubMed1978

External validity in the assessment of intellectual development in adulthood.

The relation of intelligence and competence is discussed and external validity issues are examined for the dimensions of settings, measurement variables, treatment variables and experimental units. It is argued that external validity across situations and life stages cannot be obtained for any single measure of intellectual ability. External validity problems are exacerbated beyond young adulthood since single criterion goals comparable to that of educational aptitude in work with the young are not available, and tasks do not retain ecological validity when the situational context of the individual under study changes due to developmental progression and idiosyncratic modification of individual life situation and roles. External validity in adulthood must therefore be addressed by examining task-by-person-by-situation interfaces separately for different life stages and across cohort groupings. A major test construction and validation program is outlined, and examples are given showing how some of the aspects of such a program can be operationalized.

Cognition↗

PubMed2001

Establishing the internal and external validity of experimental studies.

The information needed to determine the internal and external validity of an experimental study is discussed. Internal validity is the degree to which a study establishes the cause-and-effect relationship between the treatment and the observed outcome. Establishing the internal validity of a study is based on a logical process. For a research report, the logical framework is provided by the report's structure. The methods section describes what procedures were followed to minimize threats to internal validity, the results section reports the relevant data, and the discussion section assesses the influence of bias. Eight threats to internal validity have been defined: history, maturation, testing, instrumentation, regression, selection, experimental mortality, and an interaction of threats. A cognitive map may be used to guide investigators when addressing validity in a research report. The map is based on the premise that information in the report evolves from one section to the next to provide a complete logical description of each internal-validity problem. The map addresses experimental mortality, randomization, blinding, placebo effects, and adherence to the study protocol. Threats to internal validity may be a source of extraneous variance when the findings are not significant. External validity is addressed by delineating inclusion and exclusion criteria, describing subjects in terms of relevant variables, and assessing generalizability. By using a cognitive map, investigators reporting an experimental study can systematically address internal and external validity so that the effects of the treatment are accurately portrayed and generalization of the findings is appropriate.

Double-Blind Method↗

PubMedDate not supplied

External validity of randomised controlled trials: "to whom do the results of this trial apply?".

In making treatment decisions, doctors and patients must take into account relevant randomised controlled trials (RCTs) and systematic reviews. Relevance depends on external validity (or generalisability)--ie, whether the results can be reasonably applied to a definable group of patients in a particular clinical setting in routine practice. There is concern among clinicians that external validity is often poor, particularly for some pharmaceutical industry trials, a perception that has led to underuse of treatments that are effective. Yet researchers, funding agencies, ethics committees, the pharmaceutical industry, medical journals, and governmental regulators alike all neglect external validity, leaving clinicians to make judgments. However, reporting of the determinants of external validity in trial publications and systematic reviews is usually inadequate. This review discusses those determinants, presents a checklist for clinicians, and makes recommendations for greater consideration of external validity in the design and reporting of RCTs.

Data Interpretation, Statistical↗