Lessons from the NHS National Programme for IT.
Explore the source record for details and available documents.
Humans↗
Biomedical subjects
Enrico W Coiera
Publications and source records attributed to Enrico W Coiera.
What factors are associated with the integration of evidence retrieval technology into routine general practice settings?
BACKGROUND: Information retrieval systems have the potential to improve patient care but little is known about the variables which influence clinicians' uptake and use of systems in routine work. AIM: To determine which factors influenced use of an online evidence retrieval system. DESIGN OF STUDY: Computer logs and pre- and post-system survey analysis of a 4-week clinical trial of the Quick Clinical online evidence system involving 227 general practitioners across Australia. RESULTS: Online evidence use was not linked to general practice training or clinical experience but female clinicians conducted more searches than their male counterparts (mean use=14.38 searches, S.D.=11.68 versus mean use=8.50 searches, S.D.=9.99; t=2.67, d.f.=157, P=0.008). Practice characteristics such as hours worked, type and geographic location of clinic were not associated with search activity. Information seeking was also not related to participants' perceived information needs, computer skills, training nor Internet connection speed. Clinicians who reported direct improvements in patient care as a result of system use had significantly higher rates of system use than other users (mean use=12.55 searches, S.D.=13.18 versus mean use=8.15 searches, S.D.=9.18; t=2.322, d.f.=154 P=0.022). Comparison of participants' views pre- and post- the trial, showed that post-trial clinicians expressed more positive views about searching for information during a consultation (chi(2)=27.40, d.f.=4, P< or =0.001) and a significantly greater number reported seeking information between consultations as a result of having access to an online evidence system in their consulting rooms (chi(2)=9.818, d.f.=2, P=0.010). CONCLUSION: Clinicians' use of an online evidence system was directly related to their reported experiences of improvements in patient care. Post-trial clinicians positively changed their views about having time to search for information and pursued more questions during clinic hours.
Adult↗
Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes.
BACKGROUND: Online evidence retrieval systems are a potential tool in supporting evidence-based practice. Effective and tested techniques for assessing the impact of these systems on care delivery and patient outcomes are limited. PURPOSE AND METHODS: In this study we applied the critical incident (CI) and journey mapping (JM) techniques to assess the integration of an online evidence system into everyday clinical practice and its impact on decision making and patient care. To elicit incidents semi-structured interviews were conducted with 29 clinicians (13 hospital physician specialists, 16 clinical nurse consultants (CNCs)) who were experienced users of the online evidence system. Clinicians were also asked questions about how they had first used the system and how their use and experiences had changed over time. These narrative accounts were then mapped and scored using the journey mapping technique. RESULTS: Clinicians generated 85 critical incidents. Three categories of impact were identified: impact on clinical practice, impact on individual clinicians and impact on colleagues through the dissemination of information gained from the online evidence system. One quarter of these included specific examples of system use leading to improvements in patient care. Clinicians obtained an average journey mapping score of 22 out of a possible score of 36, demonstrating a good level of system integration. Average scores of doctors and CNCs were similar. However individuals with the same scores often had very different journeys in system integration. CONCLUSIONS: The CI technique provided clear examples of the way in which system use had influenced practice and care delivery. The JM technique was found to be a useful method for providing a quantification of the different ways and extent to which, clinicians had integrated system use into practice, and insights into how system use can influence organisational culture. The development of the journey mapping stages provides a structure by which the program logic of a clinical information system and its desired outcomes can be made explicit and be based upon users' experiences in everyday practice. Further work is required using this technique to assess its value as an evaluation method.
Clinical Competence↗
Should clinical software be regulated?
Explore the source record for details and available documents.
Australia↗
Do online information retrieval systems help experienced clinicians answer clinical questions?
OBJECTIVE: To assess the impact of clinicians' use of an online information retrieval system on their performance in answering clinical questions. DESIGN: Pre-/post-intervention experimental design. MEASUREMENTS: In a computer laboratory, 75 clinicians (26 hospital-based doctors, 18 family practitioners, and 31 clinical nurse consultants) provided 600 answers to eight clinical scenarios before and after the use of an online information retrieval system. We examined the proportion of correct answers pre- and post-intervention, direction of change in answers, and differences between professional groups. RESULTS: System use resulted in a 21% improvement in clinicians' answers, from 29% (95% confidence interval [CI] 25.4-32.6) correct pre- to 50% (95% CI 46.0-54.0) post-system use. In 33% (95% CI 29.1-36.9) answers were changed from incorrect to correct. In 21% (95% CI 17.1-23.9) correct pre-test answers were supported by evidence found using the system, and in 7% (95% CI 4.9-9.1) correct pre-test answers were changed incorrectly. For 40% (35.4-43.6) of scenarios, incorrect pre-test answers were not rectified following system use. Despite significant differences in professional groups' pre-test scores [family practitioners: 41% (95% CI 33.0-49.0), hospital doctors: 35% (95% CI 28.5-41.2), and clinical nurse consultants: 17% (95% CI 12.3-21.7; chi(2) = 29.0, df = 2, p < 0.01)], there was no difference in post-test scores. (chi(2) = 2.6, df = 2, p = 0.73). CONCLUSIONS: The use of an online information retrieval system was associated with a significant improvement in the quality of answers provided by clinicians to typical clinical problems. In a small proportion of cases, use of the system produced errors. While there was variation in the performance of clinical groups when answering questions unaided, performance did not differ significantly following system use. Online information retrieval systems can be an effective tool in improving the accuracy of clinicians' answers to clinical questions.
Clinical Competence↗
General practitioners' use of online evidence during consultations.
BACKGROUND: Clinicians have many unanswered questions during clinical encounters which may impact on the quality and outcomes of decisions made. Provision of online evidence at the point of care is one strategy that provides clinicians with easy access to up-to-date evidence in clinical settings to support evidence-based decision-making. AIM: To determine if and when general practitioners use an online evidence system in routine clinical practice, the type of questions for which clinicians seek evidence and the extent to which the system provides clinically useful answers. DESIGN OF STUDY: A prospective cohort study which involved a 4-week clinical trial of Quick Clinical, an online evidence system specifically designed around the needs of general practitioners. SETTING: Two hundred and twenty-seven clinicians who had a computer with Internet access in their consulting rooms. METHODS: Computer logs and survey analysis. RESULTS: One hundred and ninety-three general practitioners used the online evidence system to conduct on average 8.7 searches/month. The majority of these (81%) were conducted from consulting rooms and carried out between 9a.m. and 7p.m. (83%). The most frequent searches conducted related to diagnosis (40%) and treatment (35%). 83% of clinicians believed that Quick Clinical (QC) had the potential to improve patient care, and one in four users reported direct experience of improvements in care. In 73% of queries with clinician feedback participants reported that they were able to find clinically useful information during their routine work. CONCLUSION: General practitioners will use an online evidence retrieval system in routine practice, and report that its use improves the quality of patient care.
Adult↗
Measuring the Impact of Online Evidence Retrieval Systems using Critical Incidents & Journey Mapping.
Online evidence retrieval systems are one potential tool in supporting evidence-based practice. We have undertaken a program of research to investigate how hospital-based clinicians (doctors, nurses and allied health professionals) use these systems, factors influencing use and their impact on decision-making and health care delivery. A central component of this work has been the development and testing of a broad range of evaluation techniques. This paper provides an overview of the results obtained from three stages of this evaluation and details the results derived from the final stage which sought to test two methods for assessing the integration of an online evidence system and its impact on decision making and patient care. The critical incident and journey mapping techniques were applied. Semi-structured interviews were conducted with 29 clinicians who were experienced users of the online evidence system. Clinicians were asked to described recent instances in which the information obtained using the online evidence system was especially helpful with their work. A grounded approach to data analysis was taken producing three categories of impact. The journey mapping technique was adapted as a method to describe and quantify clinicians' integration of CIAP into their practice and the impact of this on patient care. The analogy of a journey is used to capture the many stages in this integration process, from introduction to the system to full integration into everyday clinical practice with measurable outcomes. Transcribed interview accounts of system use were mapped against the journey stages and scored. Clinicians generated 85 critical incidents and one quarter of these provided specific examples of system use leading to improvements in patient care. The journey mapping technique proved to be a useful method for providing a quantification of the ways and extent to which clincians had integrated system use into practice, and insights into how information systems can influence organisational culture. Further work is required on this technique to assess its value as an evaluation method. The study demonstrates the strength of a triangulated evidence approach to assessing the use and impact of online clinical evidence systems.
Attitude of Health Personnel↗
Clinicians' assessments of the usefulness of online evidence to answer clinical questions.
Family physicians' experience of using an online evidence retrieval system in routine clinical work was measured. Participants' assessments regarding the usefulness of the online evidence to answer their clinical questions were determined using three feedback channels: voluntary online feedback during the trial; randomized prompted feedback during a specific online evidence search; and responses in a post-trial survey. Using an analysis of the web logs, search outcome was related to patterns of searching in terms of number of searches and documents viewed. The majority of clinicians' searches were related to patient care and information retrieved was rated as useful in answering these questions. On average clinicians conducted two searches and viewed one document per clinical question. Search sessions which resulted in "completely answering" a clinical question involved significantly fewer searches than less successful searches. The three feedback channels provided different information but consistent overall results. The findings indicate that clinicians' assessments of online evidence use are linked to search patterns and online feedback is useful for measuring the effectiveness of systems.
Analysis of Variance↗
Evaluating the impact of information communication technologies on complex organizational systems: a multi-disciplinary, multi-method framework.
The health informatics research community has been undertaking work at the organization-technology intersection for some time now. However there is no one place in the literature which presents a rigorously-defined evaluation framework for use by practitioners and researchers seeking to assess the impact of information and communication technologies on organizational processes and outcomes. There are two main challenges. One is to conceptualize the design features of such an evaluation framework. The second is to specify what data will be gathered and how. This paper aims to address each of these problems.
Academic Medical Centers↗
Variation in the use of online clinical evidence: a qualitative analysis.
OBJECTIVE: To investigate factors influencing variations in clinicians' use of an online evidence retrieval system. SETTING: Public hospitals in New South Wales, Australia. METHOD: Web log analysis demonstrated considerable variation in rates of evidence use by clinicians at different hospitals. Focus groups and interviews were held with 61 staff from three hospitals, two with high rates of use and one with a low rate of use, to explore variation in evidence use. RESULTS: Differences between hospitals' and professional groups' (doctors, nurses and allied health) use of online evidence could be explained by organizational, professional and cultural factors. These included the presence of champions, organizational cultures which supported evidence-based practice (EBP), and database searching skills of individual clinicians. Staff shortages, ease of access and time taken to use the online evidence system were cited as barriers to use at the low use site, but no objective differences in these measures were found between the high and low use sites. CONCLUSION: Social and cultural factors were found to be better discriminators of high and low evidence use than technical factors.
Attitude of Health Personnel↗
Which clinical decisions benefit from automation? A task complexity approach.
OBJECTIVE: To describe a model for analysing complex medical decision making tasks and for evaluating their suitability for automation. METHOD: Assessment of a decision task's complexity in terms of the number of elementary information processes (EIPs) and the potential for cognitive effort reduction through EIP minimisation using an automated decision aid. RESULTS: The model consists of five steps: (1) selection of the domain and relevant tasks; (2) evaluation of the knowledge complexity for tasks selected; (3) identification of cognitively demanding tasks; (4) assessment of unaided and aided effort requirements for this task accomplishment; and (5) selection of computational tools to achieve this complexity reduction. The model is applied to the task of antibiotic prescribing in critical care and the most complex components of the task identified. Decision aids to support these components can provide a significant reduction of cognitive effort suggesting this is a decision task worth automating. CONCLUSION: We view the role of decision support for complex decision to be one of task complexity reduction, and the model described allows for task automation without lowering decision quality and can assist decision support systems developers.
Artificial Intelligence↗
Communication loads on clinical staff in the emergency department.
OBJECTIVE: To measure communication loads on clinical staff in an acute clinical setting, and to describe the pattern of informal and formal communication events, Australia. DESIGN: Observational study. SETTING: Two emergency departments, one rural and one urban, in New South Wales hospitals, between June and July 1999. PARTICIPANTS: Twelve clinical staff members, comprising six nurses and six doctors. MAIN OUTCOME MEASURES: Time involved in communication; number of communication events, interruptions, and overlapping communications; choice of communication channel; purpose of communication. RESULTS: 35 hours and 13 minutes were observed, and 1286 distinct communication events were identified, representing 36.5 events per person per hour (95% CI, 34.5-38.5). A third of communication events (30.6%) were classified as interruptions, giving a rate of 11.15 interruptions per hour for all subjects; 10% of communication time involved two or more concurrent conversations; and 12.7% of all events involved formal information sources such as patients' medical records. Face-to-face conversation accounted for 82%. While medical staff asked for information slightly less frequently than nursing staff (25.4% v 30.9%), they received information much less frequently (6.6% v 16.2%). CONCLUSION: Our results support the need for communication training in emergency departments and other similar workplaces. The combination of interruptions and multiple concurrent tasks may produce clinical errors by disrupting memory processes. About 90% of the information transactions observed involved interpersonal exchanges rather than interaction with formal information sources. This may put a low upper limit on the potential for improving information processes by introducing electronic medical records.
Communication↗
The impact of an online evidence system on confidence in decision making in a controlled setting.
OBJECTIVE: To examine the impact of online evidence retrieval on clinicians' decision-making confidence and to determine if this differs for experienced doctors and nurses. METHODS: A sample of 44 doctors and 31 clinical nurse consultants (CNCs) answered 8 clinical scenarios (600 scenario answers) before and after the use of online evidence resources. Clinicians rated their confidence in scenario answers and in the evidence they found using the information system. RESULTS: Prior to using online evidence, 37% of doctors and 18% of CNCs answered the scenarios correctly. These clinicians were more confident (56% very confident or confident) in their answers than those with incorrect (34%) answers. Doctors with incorrect answers prior to searching rated their confidence significantly higher than did nurses who were incorrect. After searching, both groups answered 50% of scenarios correctly. Clinicians with correct answers had greater confidence in the evidence found compared to those with incorrect answers. Doctors were more confident in evidence found confirming an initially correct answer than were nurses. More than 50% of clinicians who persisted with an incorrect answer after searching reported that they were confident or very confident in the evidence found. Clinicians who did not know scenario answers before searching placed equal confidence in evidence that led them to a correct or incorrect answer. CONCLUSIONS: The information obtained from an online evidence system influenced clinicians' confidence in their answers to the clinical scenarios. The relationship between confidence in answers and correctness is complex. Both existing knowledge and professional role were mediating factors. The finding that many clinicians placed confidence in information that led them to incorrect answers warrants further investigation.
Attitude of Health Personnel↗