AI Chatbots vs Oncologists: The Future of Medical Advice?

Dr Maurie Markman discusses the use of chatbots for answering questions about cancer from social media.
https://www.medscape.com/viewarticle/ai-chatbots-vs-oncologists-future-medical-advice-2024a1000jpa?src=soc_yt

-- TRANSCRIPT --
Hello. I’m Dr Maurie Markman, from City of Hope. I’d like to briefly discuss what I consider to be a quite interesting analysis. I think we’re going to see many more discussions about chatbots, large language models, and artificial intelligence.

I laugh a bit because I’m going to talk about social media. This is something that, obviously, in my era, when I was growing up in medicine and oncology, there was no such thing, but that’s where we are today.

The paper is titled, “Physician And Artificial Intelligence Chatbot Responses To Cancer Questions From Social Media.” This analysis was published recently in JAMA Oncology. We know now that social media obviously is incredibly relevant and is used very commonly by patients in general with medical illnesses, and in particular cancer patients asking questions — they or their families or friends, for example.

The question here was the ability of a chatbot to generate, in the words of the authors, high-quality, empathetic, and readable responses. Those are three different points, all of which are relevant. Number one, of course, is quality. It has to get it right. It’s got to be current. It’s got to be accurate information.

Second, we are talking about questions that patients and families are asking, not in the social media setting. I think it’s reasonable to assume this isn’t a researcher wanting to look up information. It may be a patient, a family member, or a friend. Of course, it’s got to be readable. It isn’t enough just to have the right answer, but does it make sense? Can I understand it? Those are the questions that these investigators propose.

They took 200 questions about cancer from public forums. They looked at the responses from six verified oncologists, so these were actual doctors who were responding. They looked at their responses from the chatbot model here. They looked at an objective score, like a scale, where 1 is poor and 5 is very good. They had two teams of oncologists who evaluated the quality, efficacy, and readability by looking at this objective scale.

The bottom line is — which is perhaps not surprising — the best AI platform was superior to the doctors in quality, empathy, and readability. This is just a snapshot, it’s a point in time, it’s some specific questions, it’s one review. I think one can argue that with the growing models — and they will continue to grow — the importance of social media, the need to get accurate information to the public and to patients and their families, with readable responses and empathetic responses, the use of this type of technology may very much fit the bill.

Transcript in its entirety can be found by clicking here:
https://www.medscape.com/viewarticle/ai-chatbots-vs-oncologists-future-medical-advice-2024a1000jpa?src=soc_yt