Biomedical subjects

John R Boulet

Publications and source records attributed to John R Boulet.

At least 19 recordsLinked to original sources

PubMed2006

Does composition medium affect the psychometric properties of scores on an exercise designed to assess written medical communication skills?

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools are ready to enter graduate training programs in the United States. The performance-based patient note exercise is specifically used to assess an examinee's ability to summarize, synthesize and interpret the data collected in a patient interview. The purpose of this study was to investigate whether choice of composition method (typing or writing) affected the psychometric properties of the scores. Using data for a 1-year period, the validity and reliability of typed and written notes was contrasted. Although the characteristics of individuals who chose whether or not to type were different, the statistical analyses indicated that, controlling for examinee ability, physician examiner ratings of the written summaries were not influenced by composition method. The results of this study suggest that the psychometric properties of the patient note scores are invariant with respect to composition method.

Adult↗

PubMed2006

Checklist content on a standardized patient assessment: an ex post facto review.

While checklists are often used to score standardized patient based clinical assessments, little research has focused on issues related to their development or the level of agreement with respect to the importance of specific items. Five physicians independently reviewed checklists from 11 simulation scenarios that were part of the former Educational Commission for Foreign Medical Graduate's Clinical Skills Assessment and classified the clinical appropriateness of each of the checklist items. Approximately 78% of the original checklist items were judged to be needed, or indicated, given the presenting complaint and the purpose of the assessment. Rater agreement was relatively poor with pairwise associations (Kappa coefficient) ranging from 0.09 to 0.29. However, when only consensus indicated items were included, there was little change in examinee scores, including their reliability over encounters. Although most checklist items in this sample were judged to be appropriate, some could potentially be eliminated, thereby minimizing the scoring burden placed on the standardized patients. Periodic review of checklist items, concentrating on their clinical importance, is warranted.

Clinical Competence↗

PubMed2006

Assessing the underlying structure of the United States Medical Licensing Examination Step 2 test of clinical skills using confirmatory factor analysis.

BACKGROUND: The purpose of the present study was to assess the fit of three factor analytic (FA) models with a representative set of United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) cases and examinees based on substantive considerations. METHOD: Checklist, patient note, communication and interpersonal skills, as well as spoken English proficiency data were collected from 387 examinees on a set of four USMLE Step 2 CS cases. The fit of skills-based, case-based, and hybrid models was assessed. RESULTS: Findings show that a skills-based model best accounted for performance on the set of four CS cases. CONCLUSION: Results of this study provide evidence to support the structural aspect of validity. The proficiency set used by examinees when performing on the Step 2 CS cases is consistent with the scoring rubric employed and the blueprint used in form assembly. These findings will be discussed in light of past research in this area.

Clinical Competence↗

PubMed2006

Modeling longitudinal performances on the United States Medical Licensing Examination and the impact of sociodemographic covariates: an application of survival data analysis.

BACKGROUND: This study models time to passing United States Medical Licensing Examination (USMLE) for the computer-based testing (CBT) start-up cohort using the Cox Proportional Hazards Model. METHOD: The number of days it took to pass Step 3 was treated as the dependent variable in the model. Covariates were: (1) gender; (2) native language (English or other); (3) medical school location (United States or other); and (4) citizenship (United States or other). RESULTS: Examinees were .59 times as likely to pass USMLE if they were trained abroad. Additionally, examinees who reported having English as their primary language and U.S. citizenship were more likely to ultimately pass USMLE. Finally, though gender was also associated with passing USMLE, its practical significance was very small. CONCLUSION: Cox regression provides a useful tool for modeling performances in a continuous delivery model. Findings suggest that passing the USMLE sequence tends to be associated with native English-speaking USMGs who also hold U.S. citizenship.

Female↗

PubMed2006

A comparison of the characteristics and examination performances of U.S. and non-U.S. citizen international medical graduates who sought Educational Commission for Foreign Medical Graduates certification: 1995-2004.

BACKGROUND: International medical graduates (IMGs) make up a substantial part of the U.S. physician workforce. Unfortunately, little is known about the demographic and performance characteristics of these doctors, especially over time. METHOD: Educational Commission for Foreign Medical Graduates (ECFMG) applicants from 1995-2004 were studied. The characteristics and examination performances of U.S. citizen and non-U.S. citizen IMGs were contrasted. RESULTS: ECFMG applications from U.S. citizen IMGs have been increasing. Compared to non-U.S. citizen IMGs, these individuals perform less well on some of the certification examinations, especially in the last 5 years, but are more likely to eventually achieve certification. Based on those medical students/graduates who took the examinations, the performance of U.S. citizen IMGs varied considerably from one medical school to another. CONCLUSION: The composition of the pool of ECFMG-certified IMGs available for graduate medical education has changed over the past 10 years. Shifts in the characteristics and abilities of IMGs are likely to have some future impact on health care delivery in the United States.

Certification↗

PubMed2006

Comparing traditional and computer-based training methods for standardized patients.

BACKGROUND: The use of computer-based training programs in medicine is widespread. To date, however, there has been little or no research conducted to explore how these self-delivered educational programs can be applied to the training of standardized patients (SPs). The purpose of this study was to compare a traditional SP training technique with one that was based on computer-based delivery of case materials and checklist-based scoring criteria. METHOD: Ten SPs were trained to portray 1 of 5 cases. The SPs were paired by case and then randomly assigned to either an experimental (computer-based training) or control (traditional trainer-based training) group. Forty medical students completed a 5-station clinical skills assessment where they encountered both computer- and trainer-trained SPs. Portrayal fidelity and documentation accuracy was assessed via videotape review of the individual encounters. RESULTS: With reference to checklist scoring discrepancies, documentation accuracy was significantly greater for those SPs who were trained using the computer-based delivery method. Likewise, based on a summary measure of portrayal fidelity, the performances of traditionally trained SPs were quite realistic but, nevertheless, less dependable than those for the computer-trained group. CONCLUSION: The use of computer-assisted educational programs to train SPs is effective and can lead to performance fidelity gains as well as improved accuracy with respect to the documentation of history taking and physical examination skills.

Analysis of Variance↗

PubMed2006

Relationships among subcomponents of the USMLE Step 2 Clinical Skills Examination, the Step 1, and the Step 2 Clinical Knowledge Examinations.

BACKGROUND: This research examined relationships between and among scores from the United States Medical Licensing Examination (USMLE) Step 1, Step 2 Clinical Knowledge (CK), and subcomponents of the Step 2 Clinical Skills (CS) examination. METHOD: Correlations and failure rates were produced for first-time takers who tested during the first year of Step 2 CS Examination administration (June 2004 to July 2005). RESULTS: True-score correlations were high between patient note (PN) and data gathering (DG), moderate between communication and interpersonal skills and DG, and low between the remaining score pairs. There was little overlap between examinees failing Step 2 CK and the different components of Step 2 CS. CONCLUSION: Results suggest that combining DG and PN scores into a single composite score is reasonable and that relatively little redundancy exists between Step 2 CK and CS scores.

Clinical Competence↗

PubMed2006

Effects of case characteristics on encounter time in a high-stakes standardized patient examination.

BACKGROUND: Data from national surveys indicate that patient characteristics could influence the time spent by physicians interviewing and assessing patients. The purposes of this investigation were to gather information regarding the relationship between encounter time and case characteristics for simulated clinical encounters and to provide evidence that the time provided to gather data was adequate. Timing data was extracted from United States Medical Licensing Examination Step 2 Clinical Skills. METHOD: To test the relative effects of case characteristics on encounter time, an analysis of variance was conducted with encounter time as the dependent variable and case characteristics as the independent variables. RESULTS: Mean encounter times were computed based on the case characteristics. Station format (history only, history and physical examination, telephone cases) predicted the most variance in encounter time (16%). CONCLUSIONS: The extent to which examination content is balanced from administration to administration ensures a mix of cases that provides adequate time limits for examinees.

Adolescent↗

PubMed2006

Educational commission for foreign medical graduates certification and specialty board certification among graduates of the Caribbean medical schools.

BACKGROUND: To determine the variability among Caribbean medical schools in the performance of their students against markers of quality. METHOD: The Educational Commission for Foreign Medical Graduates (ECFMG) database and the American Medical Association (AMA) Masterfile were linked. Analyses were restricted to ECFMG applicants, from 1980 to 2000, who attended the 12 Caribbean medical schools that produced 200 or more U.S. International Medical School Graduates (USIMGs). RESULTS: The schools produced 17,049 USIMG (48% of all USIMGs) and 5,840 non-USIMG (5% of all non-USIMGs) applicants for ECFMG certification. By school, the ECFMG certification rate ranged from 28% to 86% for USIMGs and from 27% to 82% for non-USIMGs. Specialty Board certification for graduates of these schools also varied widely. CONCLUSIONS: Medical schools in the Caribbean play an increasingly important role in supplying the U.S. workforce. There is considerable variability among them, both in the numbers of physicians they produce and the performance of their students against indicators of quality such as ECFMG and specialty Board certification.

Certification↗

PubMed2006

Survey on the clinical skills of osteopathic medical students.

As part of the standard-setting methods used by the National Board of Osteopathic Medical Examiners for its Comprehensive Osteopathic Medical Licensing Examination clinical skills performance evaluation (COMLEX-USA Level 2-PE), a self-administered survey was distributed electronically and by mail to deans of colleges of osteopathic medicine, directors of graduate medical education programs, osteopathic medical students, and experts chosen demographically to represent osteopathic physicians in the United States. Groups were asked to rate fourth-year osteopathic medical students and interns on their clinical skills and acceptable pass rates and expected pass rates on the COMLEX-USA Level 2-PE. The surveys were not used systematically to compute the passing standards but to provide additional support for their validity. The viewpoints of the deans differed from those of the students, osteopathic graduate medical education program directors, and experts regarding clinical skills proficiencies and acceptable pass rates. However, all of the groups agreed that, on average, some students and interns do not have adequate clinical skills. These results provide additional support for requiring acceptable performance on a comprehensive clinical skills examination before admission to osteopathic graduate medical education programs.

Attitude of Health Personnel↗

PubMed2005

Investigating gender-related construct-irrelevant components of scores on the written assessment exercise of a high-stakes certification assessment.

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise is specifically used to assess a candidate's ability to summarize and synthesize the data collected in a simulated patient interview. In a 1-year period, over 7700 first time takers completed the CSA, resulting in over 77,000 physician-based performance ratings. An initial pilot study indicated that, based solely on handwriting, the raters were able to correctly classify the gender of the candidate approximately 70% of the time. This result, combined with the fact that the notes are holistically scored, suggests that rating bias is possible. The purpose of this study was to investigate whether the gender of the candidate, the gender of the performing standardized patient, and the gender of the rater had any impact on scores. An analysis of covariance (ANCOVA) indicated that there was no significant interaction between candidate and rater gender. Female candidates significantly outperformed males, regardless of rater gender (p < 0.01, effect size = 0.23). The results of this study suggest that, based on rater, SP, and candidate characteristics, the validity of the PN ratings is not compromised.

Adult↗

PubMed2005

Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA).

Throughout the 40 year history of standardized patient assessments and OSCEs, there have been numerous advancements, including many that involve scoring the simulated clinical encounters. While there is no clear agreement on how examinees' performance should be documented or scored in an encounter, there is a consensus that several well-chosen SP encounters are required to produce reliable examinee scores. There also continues to be some debate as to who should do the scoring on an SP-based assessment. While logistics and cost will certainly play a role, it is probably best to use the person who is most familiar with the domain being assessed. In some instances this will be the SP; in others, an outside observer or content expert. Finally, with the growing use of OSCEs for summative purposes (e.g. certification, licensure), special attention must be paid to fairness issues. Since the same test form cannot be used day after day, examinee scores must be 'equated', taking into account the psychometric properties of scores from individual cases and individual SPs. To date, the CSA has been one of the highest-volume, high-stakes, standardized patient assessments to be developed and successfully administered. In 2003 alone, over 11 500 IMGs were tested. The early conceptual framework for this assessment was synthesized from the research endeavours of several notable individuals, including, amongst many others, Harden et al. 1975, Swanson & Stillman, 1990, Newble & Swanson, 1988, Vu et al. 1992 and Colliver, 1995. The early prototype administrations of the CSA, including many operational research studies, were supported and guided by Dr Friedman Ben-David, Friedman et al. 1991, 1993, Stillman et al. 1992, and Sutnick et al. 1993, 1995.

Clinical Competence↗

PubMed2005

U.S. citizens who graduated from medical schools outside the United States and Canada and received certification from the Educational Commission for Foreign Medical Graduates, 1983-2002.

PURPOSE: To provide a descriptive overview of international medical school graduates (IMGs), U.S. and non-U.S. citizens, who obtained their medical degrees outside of the United States and Canada, with a focus on where U.S. citizens received their medical education and how this choice has changed over time. METHOD: The study group included all IMGs (n = 143,926) certified by the Educational Commission for Foreign Medical Graduates (ECFMG) from 1983-2002. Descriptive statistics were calculated, including historical certification rates for non-U.S. citizen and U.S. citizen IMGs. For IMGs who were U.S. citizens (n = 18,762), the data were summarized by medical school and country of medical school. RESULTS: U.S. citizens who attended medical schools abroad were more likely to attend schools in Central America and the Caribbean than in any other geographic region. There was a steady decrease in the number of U.S. citizens graduating from European medical schools. Conversely, the number graduating from medical schools in India and Israel rose. Over the period studied, the regions of Africa, Oceania, and South America graduated relatively few U.S. citizens. CONCLUSIONS: From 1983-2002, U.S. citizens graduated from medical schools in Central America and the Caribbean more than any other geographic region. Studying the characteristics of medical schools in this region and their similarities to U.S. medical schools, such as a four-year curriculum, may explain why U.S. citizens are attracted to this region in large numbers. Additional studies focusing on the characteristics of medical schools that train IMGs, the performance of the graduates, and their posttraining practice patterns are warranted.

Canada↗

PubMed2005

Specialty board certification among U.S. citizen and non-U.S. citizen graduates of international medical schools.

BACKGROUND: Graduates of international medical schools (IMGs) make up approximately one-quarter of the physician population and play a key role in the provision of health care in the United States. This study investigated whether they differ from U.S. medical graduates (USMGs) in specialty board certification. METHOD: The study compared USMGs, U.S. citizen IMGs (USIMGs), and non-U.S. citizen IMGs (non-USIMGs) who graduated from medical school between 1958 and 1994 and were involved in direct patient care in 2003. RESULTS: There is variability among the specialties, but overall USMGs have the highest specialty certification rates followed by non-USIMGs, and USIMGs. Among recent medical school graduates, non-USIMGs have certification rates that are comparable to USMGs. CONCLUSIONS: IMGs have lower board-certification rates than USMGs, although a sizeable majority has achieved board certification in the specialty they practice. There are differences between non-USIMGs and USIMGs, with the former more likely to become board certified.

Certification↗

PubMed2005

Using factor analysis to evaluate checklist items.

BACKGROUND: Relatively little research has been published examining the use of factor analytic techniques to improve the psychometric qualities of performance assessments. The purpose of the current investigation is to illustrate the potential use of structural equation modeling (SEM) with scores from a standardized patient examination. METHOD: Checklist items for a single case were examined to compare two models within a SEM framework: a single, underlying "data gathering" construct and a two-factor model consisting of medical history interviewing and physical examination maneuvers. RESULTS: The chi2 statistics obtained were statistically significant for both the single factor (chi2=1645.8, p<.001) and the two-factor model, (chi2=1373.1; p<.001), indicating marginal model-data fit. A difference statistic was calculated (chi2=320.9; p<.001), suggesting that the two-factor model has better fit. CONCLUSIONS: The parameters estimated by each model could be used to evaluate checklist item performance. The results of this investigation illustrate the utility of this method in the analysis of checklist item data.

Clinical Competence↗

PubMed2005

Using a standardised patient assessment to measure professional attributes.

INTRODUCTION: Professionalism is an important topic in medical education today. While much work has focused on defining professionalism and teaching medical students the appropriate interpersonal behaviours, relatively little research has looked at meaningful ways of assessing the relevant attributes. METHOD: The Educational Commission for Foreign Medical Graduates (ECFMG) clinical skills assessment (CSA) uses standardised patients (SPs) to evaluate the readiness of graduates of international medical schools to enter accredited graduate training programmes in the USA. Doctor interpersonal skills, including professional qualities such as rapport, are evaluated as part of the CSA. Attentiveness, attitude and empathy, all facets of professional behaviour, are specifically targeted as part of the assessment. RESULTS: To date, over 35 000 candidates have been assessed, encompassing more than 370 000 individual SP encounters. Based on a 1-year cohort of examinees, the reliability of the individual professionalism-related component scores ranged from 0.61 to 0.70. Doctors who had graduated from medical school more recently, or were younger, generally obtained higher ratings. Professional qualities were only marginally related to measures of basic science and clinical science proficiency. Female candidates were rated significantly higher than male candidates on all dimensions. CONCLUSIONS: While some professional behaviours are probably best measured using formats such as surveys, self-assessment and critical incident techniques, certain aspects of the domain can be reliably and validly measured in SP examinations.

Attitude of Health Personnel↗

PubMed2005

A work-centered approach for setting passing scores on performance-based assessments.

Standardized patient examinations are being used for high-stakes decisions (e.g., graduation, licensure, and certification) with growing frequency. Concurrently, research on methods to determine the passing score for these types of performance-based assessments has increased. A wide variety of approaches have been considered in the past several years, many based on traditional techniques developed for use with multiple-choice examinations. More and more, techniques that center on review of examinee work have been employed, often resulting in the establishment of defensible, reproducible standards. The purpose of this article is to describe and evaluate a work-centered approach for determining the passing score for a performance-based assessment of clinical skills. A description of an approach to set passing scores that involves expert review of examinee work is presented. Application and evaluation of the method are illustrated using examination materials obtained from a high-stakes certification assessment.

Certification↗

PubMed2005

A simulation-based acute skills performance assessment for anesthesia training.

UNLABELLED: In an earlier study, trained raters provided reliable scores for a simulation-based anesthesia acute care skill assessment. In this study, we used this acute care skill evaluation to measure the performance of student nurse anesthetists and resident physician trainees. The performance of these trainees was analyzed to provide data about acute care skill acquisition during training. Group comparisons provided information about the validity of the simulated exercises. A set of six simulation-based acute care exercises was used to evaluate 43 anesthesia trainees (28 residents [12 junior and 16 senior] and 15 student nurse anesthetists). Six raters scored the participants on each exercise using either a detailed checklist, key-action items, or a global rating. Trainees with the most education and clinical experience (i.e., senior residents) received higher scores on the simulation scenarios, providing some evidence to support the validity of the multi-scenario assessment. Trainees varied markedly in ability depending on the content of the exercise. In general, anesthesia providers demonstrated similar aptitude in managing each of the six simulated events. Most participants effectively managed ventricular tachycardia, but postoperative events such as anaphylaxis and stroke were more difficult for all trainees to promptly recognize and treat. Training programs could use a simulation-based multiple encounter evaluation to measure provider skill in acute care. IMPLICATIONS: A trainee's skill in managing critical events can be assessed using a multiple scenario simulation-based performance evaluation.

Anesthesiology↗