Early applications of ChatGPT in medical practice, education and research

ChatGPT in medical education

A recent study evaluated the performance of ChatGPT on the United States Medical Licensing Examination (USMLE). The study revealed that ChatGPT passed all three exams (Step 1, Step 2 CK, Step 3) near the passing threshold without any previous training.⁴ On the other hand, Gilson et al state that there is a significant decrease in performance with an increased difficulty of the questions. However, the authors compare the performance of ChatGPT to a third-year medical student.⁵ Antaki et al tested ChatGPT for use in two multiple-choice question banks for the Ophthalmic Knowledge Assessment Program (OKAP) exam. The authors found similar results, with ChatGPT achieving 55.8% and 42.7% accuracy in the exams.⁶ Another study from Korea aimed at directly correlating the knowledge of Chat-GPT to that of medical students on the topic of parasitology. The authors revealed that the performance of ChatGPT was lower than medical students and concluded that ChatGPT's ability is not yet at an entirely acceptable level.⁷

Academic writing

Gao et al tested ChatGPT's ability to write academic abstracts. They included 50 abstracts from five high-impact medical journals and asked ChatGPT to produce research abstracts using provided titles and journal requirements. The authors concluded that all ChatGPT-derived abstracts were acceptably written, but only 8% of them respected the formatting requirements of the journals. 68% of the generated abstracts by ChatGPT were correctly identified by the reviewers due to the ‘vaguer’ and more ‘formulaic’ type of writing. An AI output detector showed similar results in detecting the ChatGPT-derived abstracts.⁸

In a recent article by Guo et al,⁹ the main attributes of ChatGPT's writing style are identified. The authors mention that ChatGPT writes in an organised manner and prefers a straightforward concept in the questions. Its answers are long and detailed, it shows less harmful information, and refuses to answer when it has no information about topics. However, it might ‘fabricate facts’ to give an answer, which should make users cautious regarding the professional use of ChatGPT. The authors observed that the main difference between ChatGPT's writing style and human writing is that the human writer is more subjective, colloquial and emotional, giving human abstracts a personal note.⁹ OpenAI warns that ChatGPT could sometimes ‘responds to harmful instructions’.^10,11

A recent study evaluated ChatGPT's ability to generate a literature review on the concept of the ‘digital twin’ in healthcare, asking it to paraphrase selected literature from 2020 to 2022. Although the results were promising, the iThenticate plagiarism detection tool identified many plagiarism matches.¹²

Interaction with patients and radiological reporting

Thurzo et al reviewed AI-based applications, including ChatGPT, in the dental field. They concluded that ChatGPT could facilitate the interaction between healthcare providers and patients in various ways, from analysing patient messages to personalising the communication between healthcare professionals and patients. However, they found that ChatGPT has limitations in relation to sophisticated tasks such as understanding the human anatomy.¹³ Nov et al tested ChatGPT against healthcare providers' responses to patients. They found that ChatGPT had a similar rate of correct answers compared to the providers.¹⁴ In a case study conducted by Jeblick et al,¹⁵ radiologists had the task of evaluating the quality of simplified radiology reports generated with ChatGPT. The results showed that the reports were ‘correct, complete, and not potentially harmful to patients’. However, incorrect statements and missing medical information that could potentially have led to harmful conclusions were also detected. Although this case study comprises small sample numbers, the authors emphasise the great potential of ChatGPT in radiology while also mentioning the need for further improvements.¹⁵