领先的大型语言模型在遵循间变性甲状腺癌症临床指南方面的表现：一项比较研究

摘要(中文译文)

未分化甲状腺癌（ATC）是一种罕见、侵袭性强且预后不良的恶性肿瘤。遵循美国国家综合癌症网络（NCCN）、美国甲状腺协会（ATA）和欧洲肿瘤内科学会（ESMO）的指南对于获得最佳患者结局至关重要。随着大语言模型（LLM）日益进入临床工作流程，严格评估其与既定指南的一致性十分必要。我们评估了五种领先的大语言模型针对未分化甲状腺癌临床问题生成符合指南的回答的能力。我们于2025年开展了一项比较研究，遵循TRIPOD-LLM（个体预后或诊断多变量模型透明报告规范——大语言模型）指南。根据ATA、NCCN和ESMO指南制定了70个不同复杂程度的临床问题。三位外科肿瘤学专家对每个问题进行了验证，随后评估了五种大语言模型的回答：ChatGPT 4.1、ChatGPT 5、Gemini 2.5 Pro、Claude Sonnet 4和DeepSeek R1。每个回答均采用5点李克特量表对相关性、清晰度、准确性和充分性进行评分。评分者间信度采用组内相关系数（ICC）和带有序权重的Gwet AC2进行评估。模型比较采用Kruskal-Wallis检验，并进行Dunn事后分析和Bonferroni校正。预先设定的敏感性分析排除了非盲法模型（ChatGPT 5），以确认结果的稳健性。所有四项指标均出现显著的性能差异：准确性（p = 0.007）、充分性（p = 0.003）、清晰度（p = 0.014）和相关性（p < 0.001）。Gemini 2.5 Pro的准确性中位数最高（4.5），其次是DeepSeek R1（4.4），而ChatGPT 4.1得分最低（4.0）。ICC值为0.34至0.44（差至中等），但Gwet AC2给出的估计值显著更高，为0.61至0.73（中等至高度一致），反映了评分范围受限对传统信度指标的影响。排除ChatGPT 5的敏感性分析证实了盲法模型之间的性能排序，所有四项指标的显著性均得以保持或增强。领先的大语言模型在与未分化甲状腺癌临床指南保持一致方面表现出参差不齐的能力。尽管表现最佳的模型有望作为辅助工具，但其在不同领域和复杂程度上的不一致性使其无法自主用于临床。这些模型应严格作为专家监督下的决策辅助工具。

Abstract

Anaplastic thyroid cancer (ATC) is a rare, aggressive malignancy with poor prognosis. Adherence to guidelines from the National Comprehensive Cancer Network (NCCN), American Thyroid Association (ATA), and European Society for Medical Oncology (ESMO) is critical for optimal patient outcomes. As large language models (LLMs) increasingly enter clinical workflows, rigorous evaluation of their alignment with established guidelines is essential. We evaluated five leading LLMs for their ability to generate guideline-concordant responses to clinical questions about ATC. We conducted a comparative study in 2025 following TRIPOD-LLM (Transparent Reporting of a Multivariable Model for Individual Prognosis or Diagnosis, Large Language Models) guidelines. Seventy clinical questions of varying complexity were developed from ATA, NCCN, and ESMO guidelines. Three surgical oncology experts validated each question and subsequently evaluated responses from five LLMs: ChatGPT 4.1, ChatGPT 5, Gemini 2.5 Pro, Claude Sonnet 4, and DeepSeek R1. Each response was scored for relevancy, clarity, accuracy, and adequacy on a 5-point Likert scale. Inter-rater reliability was assessed using both intraclass correlation coefficients (ICC) and Gwet's AC2 with ordinal weights. Model comparisons used the Kruskal-Wallis test with Dunn's post-hoc analysis and Bonferroni correction. A pre-specified sensitivity analysis excluding the unblinded model (ChatGPT 5) was performed to confirm robustness. Significant performance differences emerged across all four metrics: accuracy (p = 0.007), adequacy (p = 0.003), clarity (p = 0.014), and relevance (p < 0.001). Gemini 2.5 Pro achieved the highest median accuracy (4.5), followed by DeepSeek R1 (4.4), while ChatGPT 4.1 scored lowest (4.0). ICC values ranged from 0.34 to 0.44 (poor to moderate), but Gwet's AC2 yielded substantially higher estimates of 0.61 to 0.73 (moderate to substantial agreement), reflecting the impact of restricted score range on conventional reliability metrics. The sensitivity analysis excluding ChatGPT 5 confirmed the performance hierarchy among blinded models, with significance preserved or strengthened across all four metrics. Leading LLMs show variable capacity to align with ATC clinical guidelines. While top-performing models hold promise as supportive tools, their inconsistencies across domains and complexity levels preclude autonomous clinical use. These models should serve strictly as decision aids under expert supervision.

如何引用

Scientific reports 的其他文献

• 脑卒中幸存者坚持功能性运动的影响因素：基于结构方程模型的横断面研究
• 使用紧凑生物标志物和Kolmogorov-Arnold网络的可解释多模式框架改善了阿尔茨海默病的早期诊断
• 人类二元的领域特定感知分组：来自自闭症特征档案的证据
• 通过水文气象驱动因素的自我监督学习，为未计量流域的径流预测提供一个具有制度意识的框架
• 痔疮患者对手术治疗的知识、态度和做法
• 对照顾者的依恋模式调节了犬急性和慢性压力的生理指标
• 脂肪源性间充质干细胞条件培养基和肠系膜缺血再灌注中肠干细胞的增殖
• 一种可感知雾的轻量级视觉转换器架构，用于边缘设备上的实时海岸景观识别
• 使用RapidMiner工具和机器学习算法进行离心泵空化分析与预测
• 参与社区锻炼计划的帕金森病成年人和社区老年人的身体表现和DEXA衍生的身体成分：一项横断面研究