Researchers from Harvard and Beth Israel Deaconess Medical Center tested OpenAI’s next-generation “reasoning” model, o1-preview, in the emergency department triage process. The study revealed that the artificial intelligence model achieved a higher accuracy rate than doctors in real emergency room cases.

According to the study published in the journal Science, the o1-preview model was able to make a correct diagnosis in 67.1% of 76 emergency room cases. Two specialist doctors evaluated on the same cases showed accuracy of 55.3% and 50.0%, respectively.

Artificial Intelligence and Medical Collaboration

Researchers emphasize that the results obtained do not mean that artificial intelligence will replace doctors. Harvard’s Arjun Manrai said this technology has the potential to transform medicine, but more testing is needed to improve patient outcomes.

Adam Rodman, one of the doctors who participated in the study, stated that the role of artificial intelligence in medical decisions should have a legal status similar to clinical decision support tools.

Rodman argued that doctors should continue to protect their own accountability and that randomized controlled trials were essential for the reliability of the system.

The o1-preview model is designed to solve problems in structured steps, unlike standard chatbots. However, the researchers acknowledge that the model still has difficulties working with multimodal inputs such as medical imaging and audio evidence.

Yujin Potter from Berkeley University reminds us of the importance of security by drawing attention to the risks of artificial intelligence hallucinating and producing false information.

Technological Limits and Security

The o1-preview model demonstrated superior performance in more complex clinical cases compared to previous models such as ChatGPT-4. In tests involving 143 complex cases, the model was able to include the correct diagnosis in the differential list in 78.3% of cases.

The researchers stated that the model achieved a high success rate of 97.9% in suggesting helpful diagnoses. These results surpassed the 44.5% success rate of doctors who had the freedom to use search engines and standard medical resources.

Despite this, experts state that artificial intelligence still falls short in medical imaging benchmarks. It is predicted that one of the most important research areas of the next decade will be to improve the multimodal integration capabilities of these models.

The potential for artificial intelligence systems to set their own goals and manipulate the user is considered a risk factor by independent researchers. Buckley and his team argue that the models do not formally measure hallucination rates, but that the vast majority of the suggestions the model offers are useful.

While researchers confirm that the models carry the risk of hallucinations, they emphasize the importance of the “trust but verify” principle. What do you think about the use of artificial intelligence as a tool to assist doctors in medical diagnosis processes?