Benchmarking Different Natural Language Processing Models for Their Responses to Queries on Toothsupported Fixed Dental Prostheses in Terms of Accuracy and Consistency
[ X ]
Tarih
2025
Yazarlar
Dergi Başlığı
Dergi ISSN
Cilt Başlığı
Yayıncı
Erişim Hakkı
info:eu-repo/semantics/openAccess
Özet
Aim: This study aimed to evaluate the accuracy and consistency of responses generated by four different natural language processing (NLP) models to the queries on tooth-supported fixed dental prostheses. Materials and Method: Twelve open-ended questions in Turkish were created and posed to four different NLPs according to the following models: OpenAI o3 (LRM-O), OpenAI GPT 4.5 (LLM-G), DeepSeek R1 (LRM-R), and DeepSeek V3 (LLM-V) with pre- prompts in the morning, afternoon, and evening. The responses were evaluated with a holistic rubric. For accuracy assessments, the Kruskal–Wallis H test was used. Consistency between the graders’ responses was assessed using the Brennan and Prediger coefficient and the Cohen kappa coefficient. Consistency among LLMs was assessed using the Fleiss kappa and Krippendorff alpha coefficients (p < 0.05). Results: There was no statistically significant difference in accuracy between the LRM-O, LLM-G, LRM-R, and LLM-V groups (p = 0.30). The respective accuracies of LRM-O, LLM-G, LRM-R, and LLM-V were 77.7%, 50%, 66.6%, and 77.7%. In addition, the consistency among LLMs was found to be almost perfect, whereas that of LRMs was substantial. Conclusion: Within the limitations of the study, LRMs and LLMs exhibited similar accuracy. However, the consistency among LLMs was higher than that of LRMs.
Açıklama
Anahtar Kelimeler
Artificial intelligence, Dental prostheses, Treatment protocols
Kaynak
ADO Klinik Bilimler Dergisi (online)
WoS Q Değeri
Scopus Q Değeri
Cilt
14
Sayı
3
