JAMA Netw Open

AI-generated replies to patient queries receive higher satisfaction scores

October 21, 2024

This study is the first to evaluate satisfaction with AI-generated responses to patient medical questions in electronic health records (EHRs), finding higher satisfaction with AI responses compared with clinicians, though satisfaction didn't always align with clinician-rated information quality and empathy, and highlighting the need for patient involvement in AI development for optimal integration into practice.

In this cross-sectional study, researchers screened 1,089 clinical questions from 3,769,023 Patient Medical Advice Requests in EHRs, ultimately including 59 messages. Two generative AIs (ChatGPT-4 and Stanford GPT) generated responses with and without prompt engineering. Six licensed clinicians evaluated these AI and original clinician responses for information quality and empathy using a 5-point Likert scale. Additionally, 30 survey participants assessed satisfaction with AI and clinician responses. A total of 2,118 assessments were analyzed for AI response quality and 408 for satisfaction.

Findings

• Satisfaction scores were higher for AI responses (3.96) vs. clinicians’ responses (3.05), both overall (P < 0.001) and by specialty.
• AI responses to cardiology questions had the highest satisfaction (4.09), while endocrinology questions scored highest in information quality and empathy.
• Clinicians’ responses were shorter (mean 254.37 characters) than AI responses (mean 1,470.77 characters).
• Satisfaction was linked to the length of clinicians’ responses overall and in cardiovascular questions, but AI response length wasn't.

Source:

Kim J, et al. (2024, October 16). JAMA Netw Open. Perspectives on Artificial Intelligence–Generated Responses to Patient Messages. https://pubmed.ncbi.nlm.nih.gov/39412810/