Benchmarking Different Natural Language Processing Models for Their Responses to Queries on Toothsupported Fixed Dental Prostheses in Terms of Accuracy and Consistency

Basit Öğe Kaydı
dc.contributor.authorÇolpak, Emine Dilara
dc.contributor.authorYilmaz, Deniz
dc.date.accessioned2026-01-24T12:00:53Z
dc.date.available2026-01-24T12:00:53Z
dc.date.issued2025
dc.departmentAlanya Alaaddin Keykubat Üniversitesi
dc.description.abstractAim: This study aimed to evaluate the accuracy and consistency of responses generated by four different natural language processing (NLP) models to the queries on tooth-supported fixed dental prostheses. Materials and Method: Twelve open-ended questions in Turkish were created and posed to four different NLPs according to the following models: OpenAI o3 (LRM-O), OpenAI GPT 4.5 (LLM-G), DeepSeek R1 (LRM-R), and DeepSeek V3 (LLM-V) with pre- prompts in the morning, afternoon, and evening. The responses were evaluated with a holistic rubric. For accuracy assessments, the Kruskal–Wallis H test was used. Consistency between the graders’ responses was assessed using the Brennan and Prediger coefficient and the Cohen kappa coefficient. Consistency among LLMs was assessed using the Fleiss kappa and Krippendorff alpha coefficients (p < 0.05). Results: There was no statistically significant difference in accuracy between the LRM-O, LLM-G, LRM-R, and LLM-V groups (p = 0.30). The respective accuracies of LRM-O, LLM-G, LRM-R, and LLM-V were 77.7%, 50%, 66.6%, and 77.7%. In addition, the consistency among LLMs was found to be almost perfect, whereas that of LRMs was substantial. Conclusion: Within the limitations of the study, LRMs and LLMs exhibited similar accuracy. However, the consistency among LLMs was higher than that of LRMs.
dc.identifier.doi10.54617/adoklinikbilimler.1698260
dc.identifier.endpage223
dc.identifier.issn1307-3540
dc.identifier.issue3
dc.identifier.startpage215
dc.identifier.trdizinid1348559
dc.identifier.urihttps://search.trdizin.gov.tr/tr/yayin/detay/1348559
dc.identifier.urihttps://doi.org/10.54617/adoklinikbilimler.1698260
dc.identifier.urihttps://hdl.handle.net/20.500.12868/3802
dc.identifier.volume14
dc.indekslendigikaynakTR-Dizin
dc.language.isoen
dc.relation.ispartofADO Klinik Bilimler Dergisi (online)
dc.relation.publicationcategoryMakale - Ulusal Hakemli Dergi - Kurum Öğretim Elemanı
dc.rightsinfo:eu-repo/semantics/openAccess
dc.snmzKA_TR-Dizin_20260121
dc.subjectArtificial intelligence
dc.subjectDental prostheses
dc.subjectTreatment protocols
dc.titleBenchmarking Different Natural Language Processing Models for Their Responses to Queries on Toothsupported Fixed Dental Prostheses in Terms of Accuracy and Consistency
dc.typeArticle

Dosyalar

Koleksiyon

TR-Dizin İndeksli Yayınlar Koleksiyonu