Evaluating the Accuracy and Clinical Relevance of ChatGPT in Answering Patient Queries Related to Common Health Conditions

Background: The increasing reliance on artificial intelligence (AI) in healthcare has raised important questions regarding the accuracy and clinical applicability of AI-driven responses, particularly from language models like ChatGPT. Patients frequently turn to online resources for medical advice, but the quality of such information remains variable. This study aimed to evaluate the accuracy, reliability, and clinical relevance of ChatGPT’s responses to commonly asked health-related questions. Materials and Methods: A cross-sectional observational study was conducted between January and March 2025. A panel of 10 licensed health care professionals generated a dataset of 100 frequently asked patient queries covering general medicine, pediatrics, dermatology, mental & dental health. Responses were obtained from ChatGPT (GPT-4) and evaluated by three independent physicians for accuracy, clinical relevance, and completeness using a 5-point Likert scale. Inter-rater reliability was assessed using Cohen’s kappa. Results: Out of 100 responses, 86% were rated as clinically accurate (mean score 4.3±0.5), while 78% were deemed relevant to actual clinical practice. Completeness was rated high in 73% of cases. The inter-rater reliability among the evaluators was substantial (κ = 0.78). However, 9% of the responses were flagged as potentially misleading or lacking important clinical context. Conclusion: ChatGPT demonstrates a high level of accuracy and clinical relevance in addressing common patient queries, suggesting its potential as a supplementary tool in patient education. Nevertheless, caution must be exercised, especially in complex or context-specific conditions, where professional medical consultation remains essential.