1. The large language model, GPT-4, was capable of assessing clinical acuity when presented with pairs of patient histories obtained from emergency department documentation with an accuracy comparable to a physician reviewer.

Evidence Rating Level: 2 (Good)

In recent years, the interest for the utility of large language models (LLMs) in healthcare has increased, with studies showing LLMs have achieved passing scores in the United States Licensing Medical Examination and can solve various clinical diagnostic challenges. However, few studies have examined LLMs in clinical practice with data derived from real electronic health records. This cross-sectional study therefore sought to evaluate the ability of the LLM, GPT-4 (OpenAI model = “gpt-4-0314”), in determining the clinical acuity of patients presenting to the emergency department (ED) as defined by the Emergency Severity Index (ESI). All adult visits to the University of California, San Francisco’s ED between January 1st, 2012 to January 17th, 2023 with a corresponding ED clinician note were identified to create a random pool of 10,000 pairs of ED visits with nonequivalent ESI scores. These pairs were used to query GPT-4 to return the patient with the higher clinical acuity among the pairs based on their individual clinical histories on ED presentation. A subsample of 500 pairs was also manually classified by a resident physician for comparison between LLM and human accuracy. The accuracy of the LLM was 0.89 (95% CI, 0.89-0.90) across the total sample, and the LLM’s accuracy was comparable to that of the physician for the 500 pair subsample (accuracy 0.88, [95% CI, 0.86-0.91] versus accuracy, 0.86, [95% CI, 0.83-0.89]). Overall, this study showed that the LLM, GPT-4, can accurately identify the higher acuity patient presentation when given a pair of patient presentations with an accuracy comparable to a trained physician. 

Click to read the study in JAMA Network Open

Image: PD

©2024 2 Minute Medicine, Inc. All rights reserved. No works may be reproduced without expressed written consent from 2 Minute Medicine, Inc. Inquire about licensing here. No article should be construed as medical advice and is not intended as such by the authors or by 2 Minute Medicine, Inc.