ChatGPT is not accurate as a diagnostic tool but does offer some medical educational benefits, according to a study published in PLOS One. The authors investigated ChatGPT’s diagnostic accuracy and utility in medical education. The analysis included 150 case challenges from September 2021 to January 2023 inputted into ChatGPT. The researchers found that ChatGPT answered 49% of cases correctly, with an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an area under the curve of 0.66. ChatGPT struggled with interpreting laboratory values and imaging results but was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. More than half of the AI’s answers were complete and relevant (52%), and a similar percentage of answers were characterized as low cognitive load (51%).
