Biomedical subjects

L W T Schuwirth

Publications and source records attributed to L W T Schuwirth.

6 recordsLinked to original sources

PubMed2006

Is an Angoff standard an indication of minimal competence of examinees or of judges?

BACKGROUND: To establish credible, defensible and acceptable passing scores for written tests is a challenge for health profession educators. Angoff procedures are often used to establish pass/fail decisions for written and performance tests. In an Angoff procedure judges' expertise and professional skills are assumed to influence their ratings of the items during standard-setting. The purpose of this study was to investigate the impact of judges' item-related knowledge on their judgement of the difficulty of items, and second, to determine the stability of differences between judges. METHOD: Thirteen judges were presented with two sets of 60 items on different occasions. They were asked to not only judge the difficulty of the items but also to answer them, without the benefit of the answer key. For each of the 120 items an Angoff estimate and an item score were obtained. The relationship between the Angoff estimate and the item score was examined by applying a regression analysis to the 60 items (Angoff estimate, score) for each judge at each occasion. RESULTS AND CONCLUSIONS: This study shows that in standard-setting the individual judgement of the individual item is not only a reflection of the difficulty of the item but also of the inherent stringency of the judge and his/her subject-related knowledge. Considerable variation between judges in their stringency was found, and Angoff estimates were significantly affected by a judge knowing or not knowing the answer to the item. These findings stress the importance of a careful selection process of the Angoff judges when making pass/fail decisions in health professions education. They imply that judges should be selected who are not only capable of conceptualising the 'minimally competent student', but who would also be capable of answering all the items.

Clinical Competence↗

PubMed2005

[Assessment of medical competence in clinical education].

There has been considerable change in the field of assessment of medical competence. At the moment, competency-orientated assessment, 'mini-CEX' (brief clinical evaluation exercises) and portfolios are quite popular. These methods are based on research findings indicating that medical competence can better be described as a collection of the complex tasks (so-called competencies) that a doctor must be able to perform than as the sum of knowledge, skills, problem-solving ability and attitudes. Mini-CEX represents a method for the assessment of medical competence reliably and validly in a practical setting. Using a portfolio, information on the student's competence can be collated and evaluated from various sources, including mini-CEX. As such, a portfolio has much in common with a patient chart.

Clinical Competence↗

PubMed2004

Cross institutional collaboration in assessment: a case on progress testing.

The practice of assessment is governed by an interesting paradox. On the one hand good assessment requires substantial resources which may exceed the capacity of a single institution and we have reason to doubt the quality of our in-house examinations. On the other hand, our parsimonity with regard to our resources makes us reluctant to pool efforts and share our test material. This paper reports on an initiative to share test material across different medical schools. Three medical schools in The Netherlands have successfully set up a partnership for a specific testing method: progress testing. At present, these three schools collaboratively produce high-quality test items. The jointly produced progress tests are administered concurrently by these three schools and one other school, which buys the test. The steps taken in establishing this partnership are described and results are presented to illustrate the unique sort of information that is obtained by cross-institutional assessment. In addition, plans to improve test content and procedure and to expand the partnership are outlined. Eventually, the collaboration may even extend to other test formats. This article is intended to give evidence of the feasibility and exciting potential of between school collaboration in test development and test administration. Our experiences have demonstrated that such collaboration has excellent potential to combine economic benefit with educational advantages, which exceed what is achievable by individual schools.

Cooperative Behavior↗

PubMed2003

The use of clinical simulations in assessment.

CONTEXT: Simulation-based testing methods have been developed to meet the need for assessment procedures that are both authentic and well-structured. It is widely acknowledged that, although the authenticity of a procedure may be a contributing factor to its validity, authenticity alone never is a sufficient factor. AIM: In this paper we describe the mainstream development of various simulation-based approaches, with their strengths and weaknesses. The purpose is not to provide a review based on an extensive meta-analysis but to present crucial factors in the development of these methods and their implications for current and future developments. METHOD: The description of these simulation-based instruments uses a subdivision according to the layers of Miller's pyramid. Written and computer-based simulations are aimed at measuring the 'knows how' layer, observation-based techniques such as standardised patient-based examinations and objective structured clinical examinations target the 'shows how' layer and performance practice measures assess performance at the 'does' layer. CONCLUSION: In all simulations, case specificity was found to pose the most prominent threat to reliability, while too much structure threatened to trivialise the assessment. The conclusion is that authentic and reliable assessment is predicated on a wise balance between efficiency and adequate content sampling.

Clinical Competence↗

PubMed2002

Selecting performance assessment methods for experienced physicians.

BACKGROUND: While much is now known about how to assess the competence of medical practitioners in a controlled environment, less is known about how to measure the performance in practice of experienced doctors working in their own environments. The performance of doctors depends increasingly on how well they function in teams and how well the health care system around them functions. METHODS: This paper reflects the combined experiences of a group of experienced education researchers and the results of literature searches on performance assessment methods. CONCLUSION: Measurement of competence is different to measurement of performance. Components of performance could be re-conceptualised within a different domain structure. Assessment methods may be of a different utility to that in competence assessment and, indeed, of different utility according to the purpose of the assessment. An exploration of the utility of potential performance assessment methods suggests significant gaps that indicate priority areas for research and development.

Clinical Competence↗

PubMed2002

When enough is enough: a conceptual basis for fair and defensible practice performance assessment.

INTRODUCTION: An essential element of practice performance assessment involves combining the results of various procedures in order to see the whole picture. This must be derived from both objective and subjective assessment, as well as a combination of quantitative and qualitative assessment procedures. Because of the severe consequences an assessment of practice performance may have, it is essential that the procedure is both defensible to the stakeholders and fair in that it distinguishes well between good performers and underperformers. LESSONS FROM COMPETENCE ASSESSMENT: Large samples of behaviour are always necessary because of the domain specificity of competence and performance. The test content is considerably more important in determining which competency is being measured than the test format, and it is important to recognise that the process of problem-solving process is more idiosyncratic than its outcome. It is advisable to add some structure to the assessment but to refrain from over-structuring, as this tends to trivialise the measurement. IMPLICATIONS FOR PRACTICE PERFORMANCE ASSESSMENT: A practice performance assessment should use multiple instruments. The reproducibility of subjective parts should not be increased by over-structuring, but by sampling through sources of bias. As many sources of bias may exist, sampling through all of them may not prove feasible. Therefore, a more project-orientated approach is suggested using a range of instruments. At various timepoints during any assessment with a particular instrument, questions should be raised as to whether the sampling is sufficient with respect to the quantity and quality of the observations, and whether the totality of assessments across instruments is sufficient to see 'the whole picture'. This policy is embedded within a larger organisational and health care context.

Clinical Competence↗