The AI ChatGPT could pass the exam to practice as a doctor in the United States
An investigation shows that the system achieved a hit rate close to the pass rate in the USMLE tests. However, AI still has some limitations
ChatGPT continues to rack up college degrees. After recently passing graduate exams in Law and Economics conducted by the American universities of Minnesota and Pennsylvania, artificial intelligence, capable of answering virtually any question the user asks, has shown that it is close to being able to pass the tests necessary to practice as a physician in the United States.
The research, which has been published in the specialized magazine PLOS Digital Health, shows that the OpenAI chatbot, which has 100 million monthly users, was able to correctly answer about 60% of the questions with which it was compatible and that they collected within the USMLE (the United States Medical Licensing Exam). Precisely, that 60% of correct answers is the percentage that any aspiring doctor who takes the test must pass.
The system also offered answers that agreed with the question asked in 94.6% of the cases. It even outperformed PubMedGPT, a model similar to ChatGPT, though in this case made exclusively of biomedical data, which scored 50.8% on an older dataset of questions similar to those found in the USMLE. That is because the OpenAI solution is constantly being trained with large amounts of information to improve the results it is capable of offering to Internet users.
Post– Twitter earns up to $19 million for restored accounts
An investigation with limitations
Despite the good results obtained, the research shows the limitations that, for the moment, the popular AI continues to have, and all this despite the efforts that the startup behind its creation is devoting to developing and improving it.
«It has limitations such as a limited case size (it was only tested with 350 questions). It also represents a limited scenario since it only works with texts. 26 questions that contained images or other non-text information (with which ChatGPT cannot work) were eliminated”, explains Alfonso Valencia, director of Life Sciences at the Barcelona National Supercomputing Center, in statements collected by SMC Spain.
“In addition, the results of the examination carried out by ChatGPT were evaluated by two doctors. In this way, it is necessary to wait for subsequent studies that can have a greater number of qualified evaluators to be able to endorse the results of this AI”, points out, for her part, Lucía Ortiz, a researcher in Ethics and Governance of artificial intelligence at the University Autonomous of Madrid.
Post– Bill Gates: “ChatGPT will change the world”
AI is not infallible
Despite the good work that the conversational AI developed by OpenAI is capable of doing in various situations, experts and researchers warn about the danger of believing at face value all those answers it offers. “Over the last 12 months, there has been a huge leap in the big natural language processing models, like ChatGPT. They are much better trained than what we had before,” says José del Barrio, head of the startup dedicated to developing solutions based on AI Samaipata.
“It is very easy for them to get out of control and offer information that is not true or that can be controversial. There is a risk even if they are very good at answering questions and may seem very coherent, ”says the expert.
Studies such as the one that shows that ChatGPT is close to passing an exam to become a doctor, albeit cheating because it is unable to answer all the questions, show that the tool continues to improve by leaps and bounds and even surpasses other natural language models. exclusively trained with biomedical information.
The research also highlights the capabilities of artificial intelligence as a support tool, in this case, accessible to professionals who practice medicine. Now be clear that not everything it says is correct and that the information must be checked to avoid making mistakes.