According to recent estimates, sepsis is attributable to approximately one in three hospital deaths in the United States. “Sepsis is a common and deadly condition that can be difficult to identify in the hospital because it’s a relatively heterogeneous condition and not always obvious,” explains Gabriel Wardi, MD, MPH, FACEP. “Early identification of, and treatment for, sepsis with intravenous antibiotics, fluids to correct low blood pressure, and/or surgery, if necessary, can decrease mortality and reduce time spent in the hospital.”
The increased adoption of EHRs in hospitals has led to the development of machine learning-based surveillance tools to detect and predict sepsis. “There’s great interest from healthcare professionals and administrators to use artificial intelligence (AI) to improve patient-centered outcomes and reduce costs,” says Shamim Nemati, Ph.D. “However, AI researchers and implementation scientists are still working out the details that would ensure patient safety and effective workflow integration of such tools.”
Previous research suggests that a lack of AI generalizability across institutions, high false alarm rates, and risk of automation bias are barriers to the widespread adoption of sepsis prediction systems. “Alerts from AI systems are sometimes ‘false positives,’ which can lead to increased cognitive load on the already busy caregivers and, even worse, inappropriate antibiotic treatment and distraction from other impending life-threatening issues facing the patients,” Dr. Nemati says. “Ultimately, false alerts can cause mistrust in AI systems and can potentially sabotage promising methods for the early detection of sepsis.”
COMPOSER Helps Identify, Prioritize High-Risk Patients Earlier
Dr. Wardi, Dr. Nemati, and colleagues conducted a study, published in NPJ Digital Medicine, that reported on COMPOSER, a proposed deep learning model for predicting the onset of sepsis 4-48 hours prior to the time of clinical suspicion. COMPOSER (Conformal Multidimension Prediction of Sepsis Risk) was designed to reduce false alarms by detecting unfamiliar patients or situations that can arise from erroneous or missing data, distributional shifts, and data drifts (Table). “We wanted to develop an AI system that minimizes false alerts by recognizing when the algorithm cannot reliably make a prediction,” says Dr. Nemati. “This is akin to a physician saying, ‘I’m not sure’ and then waiting for more data before making a definitive assessment and starting a treatment plan,” adds Dr. Wardi.
Instead of making spurious predictions, COMPOSER identified and flagged 20% of non-septic episodes as indeterminate, compared with a rate of 8% for septic patients, and overall, achieved a 75% to 85% reduction in false alarms. “Our study addressed major concerns that many clinicians have with AI algorithms,” says Dr. Wardi. “The algorithm significantly decreased the number of false alerts in patients with sepsis. The real benefit is that clinicians can rely on the AI system as a second pair of eyes to detect sepsis much earlier and start therapies that we know can save lives without worrying about a low-performing, untrustworthy AI system. In effect, the AI algorithm increases confidence that the sepsis alerts received are likely real and should be acted upon.” Dr. Nemati adds that reductions in false alarms did not come at the cost of reducing model sensitivity. “We reduced false alarm rates while also catching true sepsis cases by paying extra attention to the quality of input data that is required for making reliable predictions,” he notes.
Broad Applicability to Different Care Settings
Dr. Wardi says the study team focused on what makes algorithms generalizable across different healthcare systems. “Most existing AI algorithms do not have a well-defined ‘condition for use’ and ‘expiration data,’” he explains. “The FDA advocates for the design of algorithm change protocols (ACPs), which include detailed plans for updating models safely and effectively. This is important because healthcare systems are living ecosystems that evolve over time. Our work takes a major step toward monitoring the quality of data and model performance while establishing safe and effective ACPs for model updates.”
With support from the UC San Diego Health Information Services, the study team developed an interoperable and portable software platform (relying on HL7 FHIR interoperability standards and a secure HIPAA cloud) and have integrated sepsis risk scores into their EHR system. Clinical trials are planned to validate COMPOSER’s predictions in a real-time clinical setting. “We’re actively testing the system from the perspective of usability and clinical workflow integration,” says Dr. Wardi. “We’re confident this innovation will result in wider adoption of AI alerts by bedside caregivers, earlier detection of sepsis, and, ultimately, lives saved. Prospective randomized trials on healthcare AI are uncommon, but we hope these studies will further demonstrate the impact of AI on improving standards of care.”
