Reliability of an AI-based Conversational Agent in Pediatric Dentofacial Health Counseling

Abstract

Objective: To evaluate the reliability and quality of Grok artificial intelligence’s (AI) responses to commonly asked questions about pediatric dentofacial health.
Methods: Four queries related to detrimental oral habits were submitted to Grok AI in separate sessions. The generated responses were independently evaluated by orthodontists, pediatricians, and parents without a medical or dental background. Reliability was assessed using the eight-item modified DISCERN instrument, and overall information quality was evaluated using the Global Quality Scale (GQS). Group comparisons were performed using the Kruskal-Wallis test, and correlations among scores were analyzed using Spearman’s rank correlation.
Results: Across evaluator groups, mean modified DISCERN scores ranged from 23.85 to 27.66, and mean GQS scores ranged from 3.61 to 4.12, corresponding, overall, to moderate-to-good reliability and quality. No statistically significant differences were observed among orthodontists, pediatricians, and parents for any query. Correlation analyses showed significant positive associations among several modified DISCERN and GQS parameters, particularly across different queries within the same evaluator groups. These findings suggested consistent scoring patterns, especially among parents and pediatricians.
Conclusion: Grok AI provided moderately reliable and generally useful responses regarding pediatric dentofacial health. However, its responses should be considered as supportive educational information rather than a substitute for a professional pediatric or orthodontic assessment. Further validation using larger question sets, multiple AI models, and more diverse evaluator groups is needed.