Biomedical subjects

John R Boulet

Publications and source records attributed to John R Boulet.

At least 37 records · Page 2Linked to original sources

PubMed2004

Detecting score drift in a high-stakes performance-based assessment.

Although studies have been conducted to examine the effects of a variety of factors on the comparability of scores obtained from standardized patient examinations (SPE), little research has been conducted to specifically investigate the challenge of detecting drift in case difficulty estimates over time, particularly for large-scale, performance-based, assessments. The purpose of the current study was to investigate the use of a procedure to detect drift in the difficulty estimates for a large-scale, high stakes SPE. The results of this investigation suggest that, for particular performance tasks, there was some variation in mean scores over time. These findings indicate that, although it is feasible to create a bank of case-SP means and link scores back to these fixed estimates, special attention must be paid to the standardization of exam materials over time. This is essential to ensure comparability of scores and pass-fail decisions for candidates who are assessed on multiple test forms throughout the year.

Clinical Competence↗

PubMed2004

Assessing the written communication skills of medical school graduates.

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN ratings can be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.

Cohort Studies↗

PubMed2004

Acute care skills in anesthesia practice: a simulation-based resident performance assessment.

BACKGROUND: A recurring initiative in graduate education is to find more effective methods to assess specialists' skills. Life-sized simulators could be used to assess the more complex skills expected in specialty practice if a curriculum of relevant exercises were developed that could be simply and reliably scored. The purpose of this study was to develop simulation exercises and associated scoring methods and determine whether these scenarios could be used to evaluate acute anesthesia care skills. METHODS: Twenty-eight residents (12 junior and 16 senior) managed three intraoperative and three postoperative simulation exercises. Trainees were required to make a diagnosis and intervention in a simulation encounter designed to recreate an acute perioperative complication. The videotaped performances were scored by six raters. Three raters used a checklist scoring system. Three faculty raters measured when trainees performed three key diagnostic or therapeutic actions during each 5-min scenario. These faculty also provided a global score using a 10-cm line with scores from 0 (unsatisfactory) to 10 (outstanding). The scenarios included (1) intraoperative myocardial ischemia, (2) postoperative anaphylaxis, (3) intraoperative pneumothorax, (4) postoperative cerebral hemorrhage with intracranial hypertension, (5) intraoperative ventricular tachycardia, and (6) postoperative respiratory failure. RESULTS: The high correlation among all of the scoring systems and small variance among raters' scores indicated that all of the scoring systems measured similar performance domains. Scenarios varied in their overall difficulty. Even though trainees who performed well on one exercise were likely to perform well in subsequent scenarios, the authors found that there were considerable differences in case difficulty. CONCLUSION: This study suggests that simulation can be used to measure more complex skills expected in specialty training. Similar to other studies that assess a broad content domain, multiple encounters are needed to estimate skill effectively and accurately.

Acute Disease↗

PubMed2004

Assessing the impact of examinee and standardized patient ethnicity on test scores in a large-scale clinical skills examination: gathering evidence for the consequential aspect of validity.

PURPOSE: The purpose of this study was to assess whether the interaction of examinee and standardized patient (SP) ethnicity has an impact on data gathering and written communication scores in a large-scale clinical skills assessment used for certification purposes. METHOD: The sample that was the focus of the present investigation was selected from the population of 9,551 international medical graduates (IMGs) who completed the Educational Commission for Foreign Medical Graduates' Clinical Skills Assessment between May 1, 2002, and May 31, 2003. Analyses of covariance were undertaken separately for four cases, adjusting for initial mean differences between candidate groups and controlling for stringency levels of SPs. Over 2,800 SP-IMG encounters were analyzed, ranging from 597 (Case 2) to 915 (Case 3). RESULTS: None of the SP ethnicity/examinee ethnicity interactions were statistically significant. CONCLUSIONS: Findings suggest that there is little advantage to be gained by encountering a SP of similar ethnic makeup. These results are discussed in light of past research undertaken to assess fairness issues with clinical skills examinations.

Black People↗

PubMed2004

The influence of ethnicity on patient satisfaction in a standardized patient assessment.

PURPOSE: To study possible differences in patient satisfaction ratings as a function of physician and patient ethnicity in a standardized patient (SP) performance-based assessment. METHOD: Data from 334,397 ECFMG Clinical Skills Assessment (CSA) simulated clinical encounters were analyzed. A between-groups analysis of covariance was conducted, with independent variables consisting of SP and candidate ethnicity. RESULTS: Although a significant interaction between SP and candidate ethnicity was found, averaged over all encounters the SPs were equally satisfied with doctors from all cohorts. In general, SPs provided higher satisfaction ratings for racially concordant pairings. Among the ethnic groups of SPs, Asians were the least satisfied. CONCLUSIONS: Satisfaction can vary as a function of the ethnicity of the SP. These results are similar to studies of actual patient encounters that show Asians as less satisfied than patients from other ethnic backgrounds. However, for assessment purposes, provided there is a reasonable mix of SPs, the effect is quite small.

Adolescent↗

PubMed2004

The effects of task sequence on examinee performance.

BACKGROUND: If the sequence of administration of encounters in performance-based examinations has an appreciable impact on examinee performance, the validity of any associated assessment decisions could be compromised. PURPOSE: The purpose of this article was to determine if the order in which encounters occurred in a multistation standardized patient assessment had an effect on examinee performance and, if so, could this be explained by examinee experience and familiarity with the assessment. METHODS: Analysis of the scores of over 11,000 examinees who took a 10-station clinical skills assessment was performed to determine trends across the cases. RESULTS: The results showed that administration sequence does have significant, albeit small, impact on examinee performance. In general, examinees perform slightly better as they proceed through the assessment, especially in the first few encounters. CONCLUSIONS: The possible reasons for this effect were difficult to establish, but it is likely that comfort with the examination format plays a significant role. For a given assessment, all examinees saw the same cases in a fixed sequence. Therefore, as long as performance gains are relatively small and consistent, it is unlikely that sequence effects compromised the fairness of the assessment.

Adult↗

PubMed2004

Assessing the ability of medical students to perform osteopathic manipulative treatment techniques.

While osteopathic and allopathic medicine share many commonalities, there are key practice-based differences that uniquely characterize the two professions. For osteopathic medicine, one such defining feature is the use of osteopathic manipulative treatment (OMT). Unfortunately, while various treatment modalities are taught in osteopathic medical schools, there has been relatively little work done to establish standardized evaluation protocols. The purpose of this investigation was to explore the use of OMT assessment in the context of a multistation standardized patient examination. Analysis of performance data from 121 fourth-year osteopathic medical students indicated that the ability to do OMT can be reliably and validly assessed using a combination of simulated patient encounters, trained osteopathic physician raters, and an objective rating tool. Additional studies that incorporate a larger sample of students and focus on modifications to the assessment tool and rating protocols are warranted.

Clinical Competence↗

PubMed2003

Quality assurance methods for performance-based assessments.

Performance assessments are subject to many potential error sources. For performance-based assessments, including standardized patient (SP) examinations, these error sources, if left unchecked, can compromise the validity and reliability of scores. Quality assurance (QA) measures, both quantitative and qualitative, can be used to ensure that candidate scores are accurate and reasonably free from measurement error. The purpose of this paper is to outline several QA strategies that can be used to identify potential content- and score-related problems with SP assessments. These approaches include case analyses and various comparisons of primary and observer scores. Specific examples from the ECFMG Clinical Skills Assessment (CSA) are used to educate the reader concerning appropriate statistical methods and legitimate data interpretations. The results presented in this investigation highlight the need for well-defined training regimes, regular feedback to those involved in rating/scoring performances, and detailed statistical analyses of all scores.

Clinical Competence↗

PubMed2003

Evaluating the spoken English proficiency of international medical graduates: detecting threats to the validity of standardised patient ratings.

PURPOSE: To investigate potential threats to the validity of the spoken English proficiency ratings provided by standardised patients (SPs) in high-stakes clinical skills examinations. METHOD: Spoken English ratings from 43 327 patient encounters were studied. These involved over 5000 candidates, 40% of whom were female and 33% of whom self-reported English to be their native language. Over 100 SPs were involved in the study, 51% of whom were female and 90% of whom were native English speakers. Possible performance differences in English ratings were studied as a function of candidate and SP gender, and as a function of candidate and SP native language (English versus all other languages). RESULTS: No significant candidate by SP gender effect was detected. There were no meaningful differences in mean English ratings as a function of SP or candidate gender. Likewise, English ratings did not vary as a function of either candidate or SP native language. While candidate mean English ratings were not associated with the native language of the SP, native English-speaking candidates did achieve significantly higher ratings. DISCUSSION: The lack of significant interaction between candidate and SP gender, and candidate and SP native language, suggests that the SPs provided unbiased English ratings. These results, combined with the expected higher English ratings given to candidates with English-speaking backgrounds, provides additional evidence to support the validity and fairness of spoken English proficiency ratings provided by standardised patients.

Adult↗

PubMed2003

Setting defensible performance standards on OSCEs and standardized patient examinations.

Recently, standardized patient assessments and objective structured clinical examinations have been used for high-stakes certification and licensure decisions. In these testing situations, it is important that the assessments are standardized, the scores are accurate and reliable, and the resulting decisions regarding competence ar equitable and defensible. For the decisions to be valid, justifiable standards, or cut-scores, must beset. Unfortunately, unlike the body of research specifically dedicated to multiple-choice examinations, relatively little research has been conducted on standard-setting methods appropriate for use with performance-based assessments. The purpose of this article is to provide the reader with some guidance on how to set defensible standards on performance assessments, especially those that utilize standardized patients in simulated medical encounters. Various methods are discussed and contrasted, highlighting the relevant strengths and weaknesses. In addition, based on the prevailing literature and research, ideas for future studies and potential augmentations to current performance-based standard setting protocols are advanced.

Certification↗

PubMed2003

Reliability and validity of a simulation-based acute care skills assessment for medical students and residents.

BACKGROUND: Medical students and residents are expected to be able to manage a variety of critical events after training, but many of these individuals have limited clinical experiences in the diagnosis and treatment of these conditions. Life-sized mannequins that model critical events can be used to evaluate the skills required to manage and treat acute medical conditions. The purpose of this study was to develop and test simulation exercises and associated scoring methods that could be used to evaluate the acute care skills of final-year medical students and first-year residents. METHODS: The authors developed and tested 10 simulated acute care situations that clinical faculty at a major medical school expects graduating physicians to be able to recognize and treat at the conclusion of training. Forty medical students and residents participated in the evaluation of the exercises. Four faculty members scored the students/residents. RESULTS: The reliability of the simulation scores was moderate and was most strongly influenced by the choice and number of simulated encounters. The validity of the simulation scores was supported through comparisons of students'/residents' performances in relation to their clinical backgrounds and experience. CONCLUSION: Acute care skills can be validly and reliably measured using a simulation technology. However, multiple simulated encounters, covering a broad domain, are needed to effectively and accurately estimate student/resident abilities in acute care settings.

Clinical Competence↗

PubMed2003

An examination of the relationship between clinical skills examination performance and performance on USMLE Step 2.

PURPOSE: To examine the relationship between performance on a large-scale clinical skills examination (CSE) and a high-stakes multiple-choice examination. METHOD: Two samples were used: (1) 6,372 first-taker international medical graduates (IMGs); and (2) 858 fourth-year U.S. medical students. Ninety-seven percent of IMGs and 70% of U.S. students had completed Step 2. Correlations were calculated, scatter plots produced, and regression lines estimated. RESULTS: Correlations between CSE and Step 2 ranged from .16 to .38. The observed relationship between scores confirms that CSE score information is not redundant with MCQ score information. This result was consistent across samples. CONCLUSIONS: Results suggest that the CSE assesses proficiencies distinct from those assessed by current USMLE components and therefore provides evidence justifying its inclusion in the medical licensure process.

Clinical Competence↗

PubMed2003

Introduction of patient video clips into computer-based testing: effects on item statistics and reliability estimates.

PURPOSE: Using patient video clips to evaluate examinees' skills in interpreting physical examination findings is possible with computer-based testing, but the psychometric properties of video-based questions are unknown. METHOD: We developed parallel test questions incorporating video clips or text descriptions of abnormal neurologic findings and administered them to 106 fourth-year medical students finishing their Neurology Clerkship. RESULTS: Overall, video-based questions had comparable difficulty and discrimination compared to analogous text-based questions. Preliminary studies indicated similar reliability with text- and video-based questions. CONCLUSIONS: The inclusion of patient video clips in computer-based testing is feasible from technical, practical, and psychometric perspectives. Further study is needed to gather validity evidence for this novel question format.

Clinical Clerkship↗

PubMed2003

Analysis of the relationship between score components on a standardized patient clinical skills examination.

PURPOSE: This work investigated the reliability of and relationships between individual case and composite scores on a standardized patient clinical skills examination. METHOD: Four hundred ninety two fourth-year U.S. medical students received three scores [data gathering (DG), interpersonal skills (IPS), and written communication (WC)] for each of 10 standardized patient cases. mGENOVA software was used for all analyses. RESULTS: Estimated generalizability coefficients were 0.69, 0.80, and 0.70 for the DG, IPS, and WC scores, respectively. The universe-score correlation between DG and WC was high (.83); those for DG/IPS and IPS/WC were not as strong (0.51 and 0.37, respectively). Task difficulty appears to be modestly but positively related across the three scores. Correlations between the person-by-task effects for DG/IPS and DG/WC were positive yet modest. The estimated generalizability coefficient for a ten-case test using an equally weighted composite DG/WC score was 0.78. CONCLUSIONS: This work allows for interpretation of correlations between (1) proficiencies measured by multiple scores and (2) sources of error that affect those scores as well as for estimation of the reliability of composite scores. Results have important implications for test construction and test validity.

Clinical Competence↗

PubMed2003

Correlates of performance of the ECFMG Clinical Skills Assessment: influences of candidate characteristics on performance.

PURPOSE: Since 1998, over 30,000 international medical graduates (IMGs) have taken the ECFMG CSA. Although candidate background variables have been shown to predict basic and clinical science performance, little work has focused on the relationships between candidate characteristics and clinical skills performance. The purpose of this study was to investigate whether candidate characteristics were related to CSA pass/fail status. METHOD: Logistic regression analyses were conducted to explore the relationships between candidate characteristics and CSA pass/fail status for a sample of 11,690 IMGs over a two-year period. RESULTS: Results indicated that gender, English language proficiency, recent medical school graduation, and clinical science performance (USMLE Step 2) were all related to clinical skills proficiency. CONCLUSIONS: The results provide additional support for the validity of CSA pass/fail decisions.

Adult↗

PubMed2003

The effect of task exposure on repeat candidate scores in a high-stakes standardized patient assessment.

BACKGROUND: Failing candidates often make multiple attempts on licensure and certification examinations. For performance-based assessments, where available test material is often limited, overlap in examination content is frequently inevitable. PURPOSE: The purpose of this study was to investigate the performance of repeat candidates, both on new and exposed material, on a standardized patient clinical skills assessment. METHODS: Analysis of variance techniques were used to investigate the longitudinal performance of repeat candidates. Score differences for the second test session were analyzed as function of prior exposure to assessment materials. RESULTS: Although the performance of candidates improved between the first and second assessment, score increases could not be attributed to the random exposure of examination material. CONCLUSIONS: The exposure of case content on clinical skills examinations does not appear to provide any advantage, or disadvantage, to repeat test takers. This finding, although based on average scores for repeat examinees, suggests that valid assessment scores can be obtained even if there is some overlap in assessment content.

Adult↗

PubMed2003

Using National Medical Care Survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians.

Patient characteristics, chief complaints, and diagnoses can be used to specify the examination content for performance-based assessments of clinical skills. The purpose of this investigation was to explore osteopathic and allopathic medical practice patterns and to provide summary statistics that can be used to delimit potential assessment content areas for a clinical skills assessment targeted at osteopathic physicians. Analyses of the National Ambulatory Medical Care Survey indicated that the types of patients seen by osteopathic and allopathic physicians in office-based settings are somewhat different. Furthermore, the reasons that patients seek care, and accompanying diagnostic outcomes, can vary by physician type. These differences suggest that from a content perspective, a performance-based clinical skills evaluation targeted at osteopathic physicians should be characteristically different from one designed for allopathic physicians.

Clinical Competence↗

PubMed2002

The changing pool of international medical graduates seeking certification training in US graduate medical education programs.

International medical graduates (IMGs) consistently represent approximately one fourth of both the physician workforce and the graduate medical education (GME) population of the United States. To enter into accredited US GME programs, IMGs must be certified by the Educational Commission for Foreign Medical Graduates (ECFMG). Changes in the number and characteristics of those seeking certification directly affect the GME population and the future physician workforce in the United States. In July 1998, in response to concerns that IMGs might be lacking in basic clinical skills (eg, history taking, physical examinations, communicating with patients in spoken English), the ECFMG initiated a requirement that IMGs pass a clinical skills assessment (CSA) to achieve ECFMG certification. In this study we examined the pool of IMGs seeking certification, using databases reporting on all individuals beginning the certification process from 1995 through 2001. For this period, we found that the number of IMG candidates taking the Step 1 examination decreased by 45.5% (36,983 vs 16,828), and the number of IMGs registered to take Step 2 decreased by 38.1% (31 751 vs 12 122). The number of ECFMG certificates issued annually decreased, from a range of 9000 to 12,000 (1995-1998) to fewer than 6000 (1999-2001). Although the number of IMGs annually seeking and receiving certification has decreased, the quality of the applicants appears to have improved and the number of IMGs certified annually continues to adequately fill GME positions not taken by US medical graduates.

Certification↗