The following is a summary of “Deep learning-based natural language processing for detecting medical symptoms and histories in emergency patient triage,” published in the March 2024 issue of Emergency Medicine by Lee, et al.

For a study, researchers sought to explore the feasibility of using large language models (LLMs), such as GPT and BERT, for automatic clinical diagnoses in the emergency department (ED), considering the challenges of manual recording of electronic health records (EHRs) by clinicians. Specifically, the study focused on designing and validating LLMs to identify 12 medical symptoms and 2 patient histories from simulated clinician-patient conversations within 6 primary symptom scenarios encountered in emergency triage rooms.

Classification models were developed by fine-tuning BERT, a transformer-based pre-trained model. The performance of these models was analyzed using eXplainable artificial intelligence (XAI) and the Shapley additive explanation (SHAP) method. To verify the reliability of the XAI results, a Turing test was conducted, comparing them to outcomes explained by medical workers. An emergency medicine specialist evaluated the XAI results and those provided by the medical workers.

Four pre-trained LLMs were fine-tuned, and their classification performance was compared. The KLUE-RoBERTa-based model exhibited the highest performance metrics (F1-score: 0.965, AUROC: 0.893) when evaluated on human-transcribed script data. XAI results using SHAP demonstrated an average Jaccard similarity of 0.722 compared with explanations provided by medical workers for 15 samples. The Turing test showed a small 6% discrepancy between XAI and medical workers, with mean scores of 3.327 and 3.52, respectively.

The study underscored the potential of LLMs for automating EHR recording in Korean EDs. The KLUE-RoBERTa-based model showed superior classification performance, and XAI using SHAP offered reliable explanations for model outputs. The validity of the explanations was confirmed through a Turing test, suggesting the promising application of LLMs in enhancing efficiency and accuracy in clinical settings.

Reference: sciencedirect.com/science/article/abs/pii/S0735675723006770