More than 80% of adults refer to the internet for health information, including cancer information. Within online spaces, the use of chatbots and other artificial intelligence (AI) systems such as ChatGPT has grown in recent years. Given the concerns of health misinformation and miscommunication, a group of researchers from US academic institutions conducted a small-scale study to evaluate ChatGTP accuracy of information regarding cancer myths and misconceptions.
ChatGPT uses natural language processing to generate responses to user input. To test its accuracy, the researchers asked ChatGPT for answers to 13 questions that the National Cancer Institute (NCI) considers to be common points of confusion among the general public. They then asked a panel of scientific experts to evaluate the blinded answers from both ChatGPT and the NCI for accuracy. All reviewers rated all 13 responses from NCI and 11 out of the 13 ChatGPT responses as accurate, with very few noticeable differences in the number of words or readability between the 2 sources. While generally considering the answers from ChatGPT to be accurate, some of the reviewers noted that the responses often used terms associated with hedging or uncertainty that could lead to the adoption of a belief or practice that could be harmful.
Overall, the authors concluded that ChatGPT and other AI systems have potentially positive capabilities in the context of cancer misinformation. However, there remains some skepticism as to the utility of AI regarding more specific cancer information. Oncology is an ever-changing field, and since ChatGPT is currently limited to data gathered prior to 2021, its accuracy may be somewhat outdated for cancer-specific information.
High level
As with all medical communications, the sites, sources, and algorithms of AI tools need to be monitored and regulated to ensure the information shared is accurate and not misleading. Future evaluations are needed to assess the consistency of ChatGPT accuracy in response to more diverse questions about cancer and its treatment.
Ground level
Clinicians are advised to discuss AI and ChatGPT accuracy proactively with patients who may be using AI to obtain information about their disease. While some information they receive—especially for general questions—may be accurate, the current lack of regulation and monitoring could lead to patients receiving misinformation, with potentially harmful effects.
