In a recent study published in the journal PLOS Digital Health, a team of researchers tested ChatGPT’s performance on the United States Medical Licensing Examination (USMLE).
This exam consists of three highly regulated and standardized tests, which assess knowledge in a wide range of medical disciplines, and is a requirement for medical licensure in the North American country.
ChatGPT demonstrates the potential of AI in the medical field
ChatGPT is one of the most popular resources of recent times, drawing on the resources of an advanced artificial intelligence known as a Large Language Model (LLM), designed to generate human-like text within a conversational interface by predicting sequences of words. Unlike most chatbots, ChatGPT does not have access to the Internet and generates its texts from previously learned word relationships.
The authors tested ChatGPT on 350 of the 376 public questions available on the June 2022 Medical Licensing Examination, after excluding only image-based questions. After removing uncertain answers, ChatGPT scored between 52.4% and 75.0% on all three USMLE exams, exceeding the passing threshold of approximately 60%. ChatGPT also demonstrated 94.6% agreement on its responses and produced at least one meaningful idea in 88.9% of its responses.
“Achieving the passing score for this notoriously difficult expert test, and doing so without any human reinforcement, marks a remarkable milestone in the clinical maturation of AI,” the authors of this study said in their report of results.
The researchers noted that, although the size of the input was limited, these findings suggest that ChatGPT has great potential to improve medical education and, ultimately, clinical practice. In fact, physicians at AnsibleHealth, a telemedicine platform involved in this study, are already using ChatGPT to simplify jargon-crammed clinical reports to improve understanding for the patients receiving them.
Along with appreciating the findings of this study, Dr. Tiffany Kung, a member of the team behind this research, noted that ChatGPT was not only the subject of the study, but also contributed significantly to the writing of the manuscript in which the results were reported, interacting with the authors as a colleague.