Biomedical subjects

Danette W McKinley

Publications and source records attributed to Danette W McKinley.

At least 19 recordsLinked to original sources

PubMed2006

Does composition medium affect the psychometric properties of scores on an exercise designed to assess written medical communication skills?

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools are ready to enter graduate training programs in the United States. The performance-based patient note exercise is specifically used to assess an examinee's ability to summarize, synthesize and interpret the data collected in a patient interview. The purpose of this study was to investigate whether choice of composition method (typing or writing) affected the psychometric properties of the scores. Using data for a 1-year period, the validity and reliability of typed and written notes was contrasted. Although the characteristics of individuals who chose whether or not to type were different, the statistical analyses indicated that, controlling for examinee ability, physician examiner ratings of the written summaries were not influenced by composition method. The results of this study suggest that the psychometric properties of the patient note scores are invariant with respect to composition method.

Adult↗

PubMed2006

The medical education of United States citizens who train abroad.

BACKGROUND: Graduates of international medical schools (IMGs) make up approximately one quarter of the physician workforce in the United States. Among IMGs are a number of US citizens (USIMGs) who take graduate training positions and ultimately practice in the United States. Compared with graduates of US medical schools (USMGs), relatively little is known about the undergraduate educational experiences of these US citizens. The objective of this study was to identify the schools that produce the most USIMGs and to describe the educational experiences and examination performance of graduates of these schools. METHODS: The 10 largest schools were identified based on the number of USIMGs who were certified by the Educational Commission for Foreign Medical Graduates between 2001 and 2004. Information about the institutions was collected from the International Medical Education Directory; educational experience information was gathered from 100 randomly selected transcripts and from a survey that was completed by 418 graduates of the schools. These data were compared with information about USMGs from the curriculum management and information tool and the graduation questionnaire of the Association of American Medical Colleges. Performance on steps 1 and 2CK of the United States Medical Licensing Examination were also analyzed. RESULTS: Some differences existed between USIMGs and USMGs in the required clinical clerkships, the sites where educational experiences occurred, and the special topics that were covered. USMGs and non-US citizen IMGs had better examination scores than USIMGs. CONCLUSION: There are many similarities; however, there are some noteworthy differences between the educational experiences of USIMGs and USMGs. Further work is needed to better understand the educational experiences of USIMGs, particularly in the clinical clerkships.

Clinical Clerkship↗

PubMed2006

A comparison of the characteristics and examination performances of U.S. and non-U.S. citizen international medical graduates who sought Educational Commission for Foreign Medical Graduates certification: 1995-2004.

BACKGROUND: International medical graduates (IMGs) make up a substantial part of the U.S. physician workforce. Unfortunately, little is known about the demographic and performance characteristics of these doctors, especially over time. METHOD: Educational Commission for Foreign Medical Graduates (ECFMG) applicants from 1995-2004 were studied. The characteristics and examination performances of U.S. citizen and non-U.S. citizen IMGs were contrasted. RESULTS: ECFMG applications from U.S. citizen IMGs have been increasing. Compared to non-U.S. citizen IMGs, these individuals perform less well on some of the certification examinations, especially in the last 5 years, but are more likely to eventually achieve certification. Based on those medical students/graduates who took the examinations, the performance of U.S. citizen IMGs varied considerably from one medical school to another. CONCLUSION: The composition of the pool of ECFMG-certified IMGs available for graduate medical education has changed over the past 10 years. Shifts in the characteristics and abilities of IMGs are likely to have some future impact on health care delivery in the United States.

Certification↗

PubMed2006

Effects of case characteristics on encounter time in a high-stakes standardized patient examination.

BACKGROUND: Data from national surveys indicate that patient characteristics could influence the time spent by physicians interviewing and assessing patients. The purposes of this investigation were to gather information regarding the relationship between encounter time and case characteristics for simulated clinical encounters and to provide evidence that the time provided to gather data was adequate. Timing data was extracted from United States Medical Licensing Examination Step 2 Clinical Skills. METHOD: To test the relative effects of case characteristics on encounter time, an analysis of variance was conducted with encounter time as the dependent variable and case characteristics as the independent variables. RESULTS: Mean encounter times were computed based on the case characteristics. Station format (history only, history and physical examination, telephone cases) predicted the most variance in encounter time (16%). CONCLUSIONS: The extent to which examination content is balanced from administration to administration ensures a mix of cases that provides adequate time limits for examinees.

Adolescent↗

PubMed2006

Educational commission for foreign medical graduates certification and specialty board certification among graduates of the Caribbean medical schools.

BACKGROUND: To determine the variability among Caribbean medical schools in the performance of their students against markers of quality. METHOD: The Educational Commission for Foreign Medical Graduates (ECFMG) database and the American Medical Association (AMA) Masterfile were linked. Analyses were restricted to ECFMG applicants, from 1980 to 2000, who attended the 12 Caribbean medical schools that produced 200 or more U.S. International Medical School Graduates (USIMGs). RESULTS: The schools produced 17,049 USIMG (48% of all USIMGs) and 5,840 non-USIMG (5% of all non-USIMGs) applicants for ECFMG certification. By school, the ECFMG certification rate ranged from 28% to 86% for USIMGs and from 27% to 82% for non-USIMGs. Specialty Board certification for graduates of these schools also varied widely. CONCLUSIONS: Medical schools in the Caribbean play an increasingly important role in supplying the U.S. workforce. There is considerable variability among them, both in the numbers of physicians they produce and the performance of their students against indicators of quality such as ECFMG and specialty Board certification.

Certification↗

PubMed2005

Investigating gender-related construct-irrelevant components of scores on the written assessment exercise of a high-stakes certification assessment.

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise is specifically used to assess a candidate's ability to summarize and synthesize the data collected in a simulated patient interview. In a 1-year period, over 7700 first time takers completed the CSA, resulting in over 77,000 physician-based performance ratings. An initial pilot study indicated that, based solely on handwriting, the raters were able to correctly classify the gender of the candidate approximately 70% of the time. This result, combined with the fact that the notes are holistically scored, suggests that rating bias is possible. The purpose of this study was to investigate whether the gender of the candidate, the gender of the performing standardized patient, and the gender of the rater had any impact on scores. An analysis of covariance (ANCOVA) indicated that there was no significant interaction between candidate and rater gender. Female candidates significantly outperformed males, regardless of rater gender (p < 0.01, effect size = 0.23). The results of this study suggest that, based on rater, SP, and candidate characteristics, the validity of the PN ratings is not compromised.

Adult↗

PubMed2005

Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA).

Throughout the 40 year history of standardized patient assessments and OSCEs, there have been numerous advancements, including many that involve scoring the simulated clinical encounters. While there is no clear agreement on how examinees' performance should be documented or scored in an encounter, there is a consensus that several well-chosen SP encounters are required to produce reliable examinee scores. There also continues to be some debate as to who should do the scoring on an SP-based assessment. While logistics and cost will certainly play a role, it is probably best to use the person who is most familiar with the domain being assessed. In some instances this will be the SP; in others, an outside observer or content expert. Finally, with the growing use of OSCEs for summative purposes (e.g. certification, licensure), special attention must be paid to fairness issues. Since the same test form cannot be used day after day, examinee scores must be 'equated', taking into account the psychometric properties of scores from individual cases and individual SPs. To date, the CSA has been one of the highest-volume, high-stakes, standardized patient assessments to be developed and successfully administered. In 2003 alone, over 11 500 IMGs were tested. The early conceptual framework for this assessment was synthesized from the research endeavours of several notable individuals, including, amongst many others, Harden et al. 1975, Swanson & Stillman, 1990, Newble & Swanson, 1988, Vu et al. 1992 and Colliver, 1995. The early prototype administrations of the CSA, including many operational research studies, were supported and guided by Dr Friedman Ben-David, Friedman et al. 1991, 1993, Stillman et al. 1992, and Sutnick et al. 1993, 1995.

Clinical Competence↗

PubMed2005

Specialty board certification among U.S. citizen and non-U.S. citizen graduates of international medical schools.

BACKGROUND: Graduates of international medical schools (IMGs) make up approximately one-quarter of the physician population and play a key role in the provision of health care in the United States. This study investigated whether they differ from U.S. medical graduates (USMGs) in specialty board certification. METHOD: The study compared USMGs, U.S. citizen IMGs (USIMGs), and non-U.S. citizen IMGs (non-USIMGs) who graduated from medical school between 1958 and 1994 and were involved in direct patient care in 2003. RESULTS: There is variability among the specialties, but overall USMGs have the highest specialty certification rates followed by non-USIMGs, and USIMGs. Among recent medical school graduates, non-USIMGs have certification rates that are comparable to USMGs. CONCLUSIONS: IMGs have lower board-certification rates than USMGs, although a sizeable majority has achieved board certification in the specialty they practice. There are differences between non-USIMGs and USIMGs, with the former more likely to become board certified.

Certification↗

PubMed2005

Using factor analysis to evaluate checklist items.

BACKGROUND: Relatively little research has been published examining the use of factor analytic techniques to improve the psychometric qualities of performance assessments. The purpose of the current investigation is to illustrate the potential use of structural equation modeling (SEM) with scores from a standardized patient examination. METHOD: Checklist items for a single case were examined to compare two models within a SEM framework: a single, underlying "data gathering" construct and a two-factor model consisting of medical history interviewing and physical examination maneuvers. RESULTS: The chi2 statistics obtained were statistically significant for both the single factor (chi2=1645.8, p<.001) and the two-factor model, (chi2=1373.1; p<.001), indicating marginal model-data fit. A difference statistic was calculated (chi2=320.9; p<.001), suggesting that the two-factor model has better fit. CONCLUSIONS: The parameters estimated by each model could be used to evaluate checklist item performance. The results of this investigation illustrate the utility of this method in the analysis of checklist item data.

Clinical Competence↗

PubMed2004

Detecting score drift in a high-stakes performance-based assessment.

Although studies have been conducted to examine the effects of a variety of factors on the comparability of scores obtained from standardized patient examinations (SPE), little research has been conducted to specifically investigate the challenge of detecting drift in case difficulty estimates over time, particularly for large-scale, performance-based, assessments. The purpose of the current study was to investigate the use of a procedure to detect drift in the difficulty estimates for a large-scale, high stakes SPE. The results of this investigation suggest that, for particular performance tasks, there was some variation in mean scores over time. These findings indicate that, although it is feasible to create a bank of case-SP means and link scores back to these fixed estimates, special attention must be paid to the standardization of exam materials over time. This is essential to ensure comparability of scores and pass-fail decisions for candidates who are assessed on multiple test forms throughout the year.

Clinical Competence↗

PubMed2004

Assessing the written communication skills of medical school graduates.

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN ratings can be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.

Cohort Studies↗

PubMed2004

The influence of ethnicity on patient satisfaction in a standardized patient assessment.

PURPOSE: To study possible differences in patient satisfaction ratings as a function of physician and patient ethnicity in a standardized patient (SP) performance-based assessment. METHOD: Data from 334,397 ECFMG Clinical Skills Assessment (CSA) simulated clinical encounters were analyzed. A between-groups analysis of covariance was conducted, with independent variables consisting of SP and candidate ethnicity. RESULTS: Although a significant interaction between SP and candidate ethnicity was found, averaged over all encounters the SPs were equally satisfied with doctors from all cohorts. In general, SPs provided higher satisfaction ratings for racially concordant pairings. Among the ethnic groups of SPs, Asians were the least satisfied. CONCLUSIONS: Satisfaction can vary as a function of the ethnicity of the SP. These results are similar to studies of actual patient encounters that show Asians as less satisfied than patients from other ethnic backgrounds. However, for assessment purposes, provided there is a reasonable mix of SPs, the effect is quite small.

Adolescent↗

PubMed2004

The effects of task sequence on examinee performance.

BACKGROUND: If the sequence of administration of encounters in performance-based examinations has an appreciable impact on examinee performance, the validity of any associated assessment decisions could be compromised. PURPOSE: The purpose of this article was to determine if the order in which encounters occurred in a multistation standardized patient assessment had an effect on examinee performance and, if so, could this be explained by examinee experience and familiarity with the assessment. METHODS: Analysis of the scores of over 11,000 examinees who took a 10-station clinical skills assessment was performed to determine trends across the cases. RESULTS: The results showed that administration sequence does have significant, albeit small, impact on examinee performance. In general, examinees perform slightly better as they proceed through the assessment, especially in the first few encounters. CONCLUSIONS: The possible reasons for this effect were difficult to establish, but it is likely that comfort with the examination format plays a significant role. For a given assessment, all examinees saw the same cases in a fixed sequence. Therefore, as long as performance gains are relatively small and consistent, it is unlikely that sequence effects compromised the fairness of the assessment.

Adult↗

PubMed2003

Quality assurance methods for performance-based assessments.

Performance assessments are subject to many potential error sources. For performance-based assessments, including standardized patient (SP) examinations, these error sources, if left unchecked, can compromise the validity and reliability of scores. Quality assurance (QA) measures, both quantitative and qualitative, can be used to ensure that candidate scores are accurate and reasonably free from measurement error. The purpose of this paper is to outline several QA strategies that can be used to identify potential content- and score-related problems with SP assessments. These approaches include case analyses and various comparisons of primary and observer scores. Specific examples from the ECFMG Clinical Skills Assessment (CSA) are used to educate the reader concerning appropriate statistical methods and legitimate data interpretations. The results presented in this investigation highlight the need for well-defined training regimes, regular feedback to those involved in rating/scoring performances, and detailed statistical analyses of all scores.

Clinical Competence↗

PubMed2003

Setting defensible performance standards on OSCEs and standardized patient examinations.

Recently, standardized patient assessments and objective structured clinical examinations have been used for high-stakes certification and licensure decisions. In these testing situations, it is important that the assessments are standardized, the scores are accurate and reliable, and the resulting decisions regarding competence ar equitable and defensible. For the decisions to be valid, justifiable standards, or cut-scores, must beset. Unfortunately, unlike the body of research specifically dedicated to multiple-choice examinations, relatively little research has been conducted on standard-setting methods appropriate for use with performance-based assessments. The purpose of this article is to provide the reader with some guidance on how to set defensible standards on performance assessments, especially those that utilize standardized patients in simulated medical encounters. Various methods are discussed and contrasted, highlighting the relevant strengths and weaknesses. In addition, based on the prevailing literature and research, ideas for future studies and potential augmentations to current performance-based standard setting protocols are advanced.

Certification↗

PubMed2003

Correlates of performance of the ECFMG Clinical Skills Assessment: influences of candidate characteristics on performance.

PURPOSE: Since 1998, over 30,000 international medical graduates (IMGs) have taken the ECFMG CSA. Although candidate background variables have been shown to predict basic and clinical science performance, little work has focused on the relationships between candidate characteristics and clinical skills performance. The purpose of this study was to investigate whether candidate characteristics were related to CSA pass/fail status. METHOD: Logistic regression analyses were conducted to explore the relationships between candidate characteristics and CSA pass/fail status for a sample of 11,690 IMGs over a two-year period. RESULTS: Results indicated that gender, English language proficiency, recent medical school graduation, and clinical science performance (USMLE Step 2) were all related to clinical skills proficiency. CONCLUSIONS: The results provide additional support for the validity of CSA pass/fail decisions.

Adult↗

PubMed2003

The effect of task exposure on repeat candidate scores in a high-stakes standardized patient assessment.

BACKGROUND: Failing candidates often make multiple attempts on licensure and certification examinations. For performance-based assessments, where available test material is often limited, overlap in examination content is frequently inevitable. PURPOSE: The purpose of this study was to investigate the performance of repeat candidates, both on new and exposed material, on a standardized patient clinical skills assessment. METHODS: Analysis of variance techniques were used to investigate the longitudinal performance of repeat candidates. Score differences for the second test session were analyzed as function of prior exposure to assessment materials. RESULTS: Although the performance of candidates improved between the first and second assessment, score increases could not be attributed to the random exposure of examination material. CONCLUSIONS: The exposure of case content on clinical skills examinations does not appear to provide any advantage, or disadvantage, to repeat test takers. This finding, although based on average scores for repeat examinees, suggests that valid assessment scores can be obtained even if there is some overlap in assessment content.

Adult↗

PubMed2002

Assessing the comparability of standardized patient and physician evaluations of clinical skills.

Accumulating evidence to defend decisions based on scores from evaluations is an ongoing process. The purpose of this investigation was to gather additional data to support the validity of inferences made from scores on the Educational Commission for Foreign Medical Graduates' Clinical Skills Assessment (CSA). This was accomplished by contrasting CSA candidate scores, and pass/fail decisions, with those obtained from the American Board of Internal Medicine's Mini-CEX (Clinical Evaluation Exercise). Data gathering performance based on the number of unweighted history taking and physical examination checklist items adequately predicted the global ratings provided by physician observers. CSA ratings of doctor-patient communication skills correlated with mini-CEX ratings of like constructs, indicating that physician observers, using mini-CEX rating scales, are able to make realistic assessments of interpersonal skills. These results provide evidence of the convergent validity of CSA scores.

Certification↗