Assessing validity in written tests of general practice - exploration by factor analysis of candidate response patterns to paper 1 of the MRCGP examination.
OBJECTIVE: To investigate the content validity of Paper 1 of the MRCGP examination. METHOD: Exploratory factor analysis was carried out on candidate responses to Paper 1 of the May and October MRCGP examination in 1998. Contribution of each test question across factors was assessed using a pattern matrix of the oblique rotation. Common dimensions and variations between factor sets were identified. Key testing areas were then matched against the 'domains of competence' intended to be assessed by Paper 1 (as defined within the examination blueprint matrix). RESULTS: Whilst critical appraisal, disease prevention/evidence-based medicine and clinical management emerged as areas tested consistently, content variation was observed between factor sets extracted from both sittings. In addition, some overlap, in terms of domains tested, was seen among other assessment instruments used within the examination. CONCLUSION: Paper 1 conforms to the majority of its stated intentions. However, further development of techniques for investigating validity will be required in order to minimize content variation between both sittings of the examination as well as to help more closely define areas of competence to be tested by Paper 1 of the MRCGP examination.