Contact email

gschiff@partners.org

Overview

By retrospectively analyzing software-generated outlier prescription alerts from a dataset of Partners HealthCare EHR records, the program in this study utilized machine learning to systematically evaluate the accuracy and clinical value of the MedAware software-generated alerts, helping to alleviate issues related to medication error EHR alert fatigue.

Department

Department of Medicine

Division

Division of General Internal Medicine

Collaborators

Gordon D Schiff, MD
Lynn A Volk, MHS
Mayya Volodarskaya
Deborah H Williams
Lake Walsh
Sara G Myers
David W Bates, MD, MS
Ronen Rozenblum, PhD, MPH

Status/Stage of Development

Completed

Funding Sources

MedAware, Ltd. funded this research, but were not involved in any coding development, data analysis, or manuscript preparation.

Practice Setting

Academic Medical Center

National/Policy Context

• Medication errors – such as prescribing the wrong dose of a drug or giving drugs to the wrong patient – cause significant patient morbidity and mortality.
• Prescription errors also lead to excessive  healthcare costs estimated at more than $20 billion annually in the United States. 
• Current approaches use clinical decision support systems, which only identify a small fraction of errors and suffer from high alert rates, creating “alert fatigue”.
  • Clinical decision support systems also overlook alerts related to medical concerns that may not be anticipated and programmed into the software rules.

Patient Population Served and Payor Information

• Outpatients of all ages who had at least one encounter with a Brigham & Women’s Hospital or Massachusetts General Hospital-affiliated clinician during the two year period from January 1, 2012 to December 31, 2013. 

Project Research + Planning

• The patient cohort in this retrospective study was split into two smaller groups with similar demographic traits: one used for training to generate MedAware’s individual medication models, and one used to test model performance through simulation.
• MedAware analyzed the number and types of alerts generated on an enriched sample of 300 alerts. Innovators manually selected a set of patient charts that represent the distribution of alert categories (e.g. clinical, time-dependent, and dosage outliers) across the full dataset.
  • Within each category, frequency counts were established for each alert type. MedAware then selected a random sample by identifying the 10 most frequently occurring alert types within each category.
• Once MedAware established a random sample of charts, Brigham & Women’s staff utilized study IDs and medical record numbers to identify specific patient charts for review in the EHRs.
• Patient charts were used to determine:
  • If alerts were accurate based on structured and coded information provided in the data to MedAware
  • If alerts were clinically valid based on the clinical data in the patient’s EHR.
  • If alerts were clinically useful by contributing additional information to patient care that could influence the caregiver to change medications.

Tools or Products Developed

• MedAware (Raanana, Israel) is a commercial software screening system developed to identify and prevent prescription errors.
  • MedAware uses a machine learning algorithm to analyze historical EHR data, generating a model that displays the clinical environment in which a medication is likely to be prescribed.
  • The model identifies prescriptions as significant statistical outliers given each patient’s clinical situation to be flagged as potential medication errors. 
  • Each MedAware alert includes a short description to provide user-friendly explanations allowing clinicians to understand the reason underlying the alert.
    • Clinical outliers: medication is a marked outlier based on patient characteristics (for example, prescribing birth control for an infant boy).
    • Time-dependent irregularities: changes in blood test results indicate a current medication is an outlier from a patient’s profile.
    • Dosage outliers: dosage differs greatly from the machine-learned dosage distribution of medication in the general population and in the patient’s history.

Tech Involved

• Electronic medical record

Team Members Involved

• Data Analyst

Workflow Steps

• Retrospective clinical data – including demographics, diagnoses, problem lists, outpatient and inpatient encounters, encounter clinicians, clinician specialties, procedures, medications, allergies, vital signs, and selected blood tests – was extracted from existing databases of EHR records between  January 1, 2012 to December 31, 2013 for patients included in the study.
  • Patient and clinician names, as well as medical record numbers, were removed from the dataset and replaced by random study IDs.
  • Through a secure transfer, a limited dataset was sent to MedAware for analysis.

Outcomes

• MedAware’s machine learning approach found clinically useful information regarding prescription errors –  analyzing a total of 747,985 patient records that generated 15,692 alerts in a simulation cohort consisting of 373,992 patients.
  • Those alerts represented 1706 unique alert types with an overall distribution of 29.3% clinical outliers, 66.8% time dependent, and 3.9% dosage outliers.
• 76.2% of alerts generated by MedAware’s machine learning approach were found to be valid with potential medication errors.
  • 75.0% were found to be clinically useful in flagging potential issues, with 18.8% classified with medium clinical value and 56.2% assigned high clinical value.
• The below figure displays chart reviews classified as data-related vs. clinically valid
  

Future Outcomes

• This interventional program developed a novel rating system – defining alerts as “accurate” or “valid” with levels of “clinical value” – but has not formally validated this tool, which could be valuable for use by future researchers.
• Future studies may attempt to determine precise additional benefits beyond those of existing clinical decision support systems by comparing outlier-based alerts to existing rule-based alerts.

Benefits

• Through careful evaluation, the MedAware system was found to generate potentially useful alerts with a modest rate of false positives.
• MedAware was able to generate alerts that were otherwise missed by existing clinical decision support systems with a reasonably high degree of alert usefulness when reviewing patient’s clinical contexts. 
• MedAware’s self-learning and self-adaptive capability allow it automatic and continuous search for patient and institutional-based novel outlier patterns that could represent medication errors.

Intervention-Specific Challenges

• Because Brigham & Women’s Hospital and Massachusetts General Hospital has a homegrown EHR system under Partners Healthcare, it is unclear if the findings can be generalized to other EHR systems. 
  • However, the quality of data from this study is currently stronger than that from commercial systems
• Although chart reviewers were carefully trained with a clearly developed coding manual, each chart required a degree of judgment from the reviewer and overall research team. 
• Numerous challenges exist when working with clinical data structured in EHR systems:
  • Medication start/stop dates did not always accurately reflect active prescriptions.
  • Sometimes, the care provided within Partners HealthCare was limited to a single specialty (e.g. orthopedics), and the lack of additional clinical information made it difficult to assess alert accuracy. 
  • Some diagnoses were discussed in free-text notes but not added in structured data fields, so MedAware could generate technically accurate alerts that were not valid for the clinical situation.

Sources

• Schiff, G. D., Volk, L. A., Volodarskaya, M., Williams, D. H., Walsh, L., Myers, S. G., Bates DW & Rozenblum, R. (2017). Screening for medication errors using an outlier detection system. Journal of the American Medical Informatics Association, 24(2), 281-287. https://doi.org/10.1093/jamia/ocw171