Abstract
Introduction: Infliction of unintended medical harm received increased attention. Increasing numbers of hospitals evaluate medical records of inpatients hospital death. (ref) This is labour-intensive and expensive. Information about adequacy of screening tools used for detecting adverse events (AEs) is therefore important.
Methods: We searched the literature for evidence concerning these screening methods based on the World Health Organisation (WHO) criteria for evaluating screening programs.1 Results are reported separately for the trigger tool of the Institute for Healthcare Improvement (IHI)2 and the one developed in the Harvard Medical Practice Study (HMPS).3
Results: 4353 studies were identified. After selection 57 studies with relevant information about these tests remained. The reduction in prevalence of adverse events (AE) was between 0.5%-3.5% per year. For the IHI method, the specificity ranged between 93%-100%, and the sensitivity between 34%-95%. Average positive predictive value (PPV) of the trigger systems was 38%. The Kappa (K) for agreement between doctors on the presence of an AE ranged between fair and substantial in the HMPS studies and between fair and almost perfect with IHI triggers. Average costs of an AE were €8739.
Conclusion: Retrospective medical record review has not been extensively evaluated especially not according to the WHO criteria for screening programs. There is a lack of information on improvement in AE occurrence, cost-effectiveness and the predictive value of the triggers. We recommend policy makers to scientifically evaluate the chosen methods for their utility and cost-effectiveness with an improvement of care as the end goal.
Introduction
The unintentional medical harm inflicted on patients in hospitals received increased attention during the past years. Several studies have shown significant rates of adverse events (AEs) that harm patients during their hospital stay.4-6 Therefore, interest for implementing safety- and quality programs has grown. Avoiding medical injury during hospital admission and improving patient safety, therefore, has high priority in hospitals, for healthcare inspection, and for the government.7 According to the report ‘to Err is Human’ of the Institute of Medicine,6 at least 44,000 people (and perhaps even 96,000) die each year in hospitals in the US due to possibly preventable medical errors. Fifteen years after the initial report, a recent update stresses the importance of keeping the focus on improving patient safety.6 In the Netherlands, a report by NIVEL (Dutch institute for research in healthcare) in which hospital medical records were evaluated on care related harm, increased the need for transparency and responsibility about this subject. They estimated that in Dutch hospitals every year about 1700 patients (4.1% of the total number of deaths in hospitals) die because of unintentional, but avoidable, harm. Six years after this initial report, which was published in 2007, follow up showed improvement but, 2.6% of the total number of inpatient deaths still occurred because of unintentional but preventable harm. In other countries, an incidence between 2.5% and 11.5% was found.8-12 Several reasons are thought to contribute to the risk of these AEs. First, the increasing number of elderly patients with severe comorbidity and growing medical technical possibilities to carry out more complex treatments and interventions, result in more risks. Second, due to this expansion in treatment options, the estimation of the possible health benefits is more complex. This could lead to treating more patients in which the predicted outcome is unclear, especially in the elderly and newborns with substantial comorbidities. Furthermore, social changes as part-time working and stricter adherence to prescribed working hours have their influence on the expertise and experience of medical professionals and the continuity of care. Teamwork and integrated care are therefore necessary, but they depend on good communication within and between teams which is frequently suboptimal. Also, economic limitations and cuts put health-care under pressure.13-17 All of these factors are thought to influence patient safety.
Reducing unintentional but preventable harm is important for hospitals and the methods used for detection are diverse. In 1991 the Harvard Medical Practice Study3 provided us with a powerful tool to identify cases with possible AE.18 Another method which is often used in hospitals worldwide to select cases with possible AEs is the global trigger tool (GTT), developed by the Institute for Healthcare Improvement (IHI).2 This method is found to be labour-intensive and expensive when used for reviewing all deaths in the hospital. Information about sensitivity, specificity, positive and negative predictive values of this screening tool for detecting AEs are therefore important. With this information, we can potentially determine the costs for every detected AE. However, detecting AEs is of course not the ultimate goal. Rather, the procedure aims to prevent harm to patients and even to save lives. (ref) So, adequately detecting AEs is only a small part of the whole process, which also involves feedback to the medical departments, adjustments in the delivery of care and hence improved outcome for patients. From this viewpoint, searching for AEs is actually a screening method in which the AE is the disease for which early intervention should improve the outcome.
Therefore we wondered whether these screening methods are evidence based according to the WHO criteria (box 1) for evaluating screening programs.1 Although these criteria were initially developed for the evaluation of screening programs concerning, for example, the early detection of breast malignancies or colon cancer we think they are applicable to other screening programs as well. Therefore, we searched the literature for evidence concerning the use of these trigger tools and chart review methods to identify AEs and subsequently improving patient outcome, using these criteria.
Box 1: 7 criteria for evaluating screening programs1:
Methods
Data collection
For every criterion (Box 1) we performed a literature search in Pubmed, Embase, and the Cochrane library. An overview of the search terms we used per WHO criterion is provided in table 1. We selected articles published between January 2000 and February 2016. Reviews were excluded as well as posters, comments, studies that did not concern a hospital setting and studies about adverse drug events.
Abstracts were reviewed by DK. Full articles were assessed by DK and RR. The following data were extracted if provided by the authors: Absolute number of cases reviewed, the number of (preventable) AEs, sensitivity, specificity, positive and negative predictive value (PPV and NPV), reproducibility of the methods used, the severity of AEs as well as costs. These variables were expressed as a mean with confidence intervals (CIs) when possible. We report all results for each of the WHO steps separately. Costs were not corrected for inflation. Different currencies were transformed to euros to make comparison easier. The exchange rate from June 2016 was used.
Results
The number of studies generated by every part of our search is shown in table 2. Our search provided a total of 4353 citations, of which 4334 were discarded after title and/or abstract screening, 19 studies remained for further evaluation. After reading the references in these studies we found 38 additional studies. Some studies were relevant for more than 1 step. Below, we report the results step by step.
Step 1: Our search revealed 3 suitable studies concerning an improvement in end results (table 3). In 2001, Wolff et al published about the AE rate over a certain time period concerning inpatients and patients at the emergency department.19 The absolute risk reduction for inpatients to suffer an AE was 0.61% (in a period of 8 years, on average 0.08% per year). For patients in the emergency department this was 2.78% (in a period of 2 years, on average 1.4% per year). In data from the Baylor Health Care System, Kennerly et al (2013) showed a 7% reduction in AEs in a period of 2 years (on average 3.5% per year).20 In a more recent study, Suarez et al (2014) found during a 6-year study period a decrease in absolute risk for suffering an AE of 2.5% (on average 0,4% per year).21
Step 2: Our search revealed 32 suitable studies concerning the effectiveness of the components of the multiphasic screening process. Studies were available on the trigger tool and the AE assessment strategy. We found no studies specifically addressing the result of feedback to the medical departments and the change in the rate of AEs related to this.
2.1 Trigger tool
Briefly, we distinguished two different trigger tools that were often used: the trigger tool originating from the Harvard medical practice study3,18 (HMPS) with 18 triggers and the GTT22 (IHI) with 54 triggers. Only a few studies23-25 use a different set of triggers that were not used in any other study we found. When looking at those using the HMPS triggers (figure 1), the positive predictive value (PPV), for identifying cases with AEs varied between 17 and 68% with an average of 33% (95% CI 22-44).8,10,26-35
The agreement between the nurses on the presence of a trigger showed a K value between 0.53 and 0.76 (moderate to substantial agreement)36 and an average K of 0.65 (95%CI 0.50-0.80).29,34
The IHI method (figure 1) showed a PPV between 16 and 99%, with an average of 43% (95% CI 15-71).35,37-42 There was only one study that calculated the NPV and this was 99%.37 The K in the presence of a trigger varied between 0.2039 (slight agreement) and 0.7843 (substantial agreement), which corresponds to an average of 0.57 95% CI 0.15-0.99).5,39-41,43
Studies using other trigger tools have a PPV range between 39 and 96%, with an average of 54% (95%CI 25-84).24,25,44-46 Only two studies reported a K indicating the reproducibility of the trigger system. The K varied between 0,52 and 0,68.45,47
The observed variation in these numbers was not related to the number of cases that were investigated. There was also no association with the year a study was performed.