Preliminary Evaluation of a Large Language Model-Based Tool for Complex Surgical Decision Support in Lung Cancer

This study is an exploratory effect-size estimation study, with the following specific objectives: ① to estimate the point estimate and 95% confidence interval of the Win Ratio for the experimental group (GAPS-Agent) versus the control group (large language model) in blinded pairwise preference judgments by thoracic surgery expert adjudicators, to serve as a sample size planning parameter for subsequent multicenter confirmatory clinical trials; ② to preliminarily evaluate the value of GAPS-Agent within clinical workflows.The hypothesis of this study is as follows: compared with a general-purpose large language model without medical enhancement (control group), a structured agentic workflow optimized on the basis of the GAPS evaluation framework (GAPS-Agent, experimental group) can help junior resident physicians generate clinical decision plans for complex lung cancer cases that are more strongly preferred by senior thoracic surgery expert adjudicators.

Inclusion Criteria: 1. Resident Physician Subjects: 1. Holds a valid and legally effective Physician Practice License of the People's Republic of China; 2. Currently holds the rank of resident physician in a thoracic surgery department at a tertiary Class A (3A) hospital; 3. Agrees to complete all assessment tasks of the main study phase in accordance with the study protocol; 4. Can guarantee the time and effort required to complete all assessment tasks of the main study. 2. Study Cases: 1. The case was discussed at the Thoracic Oncology Multidisciplinary Team (MDT) conference of Peking University People's Hospital between January 2025 and May 2026; 2. The current version of the NCCN guidelines does not provide an explicit recommendation covering the management of the case; 3. Does not overlap with the GAPS evaluation set; 4. The case is presented in pure text in a structured format, with all direct and indirect identifiers removed and complete de-identification performed prior to inclusion; 5. From the pool of eligible cases, 12 cases will be randomly drawn using Python (numpy.random, with a fixed and archived seed) to serve as the main study cases. The cases will cover 6 themes (chest mass of undetermined diagnosis, early-stage lung cancer, locally advanced lung cancer, oligometastatic/oligoprogressive disease, special intraoperative situations, and tumor recurrence), with 1 - 4 cases per theme. 3. Adjudication Expert Panel: 1. Holds a valid and legally effective Physician Practice License of the People's Republic of China; 2. Currently holds the rank of attending physician or above in a thoracic surgery department at a tertiary Class A hospital; 3. Chairs or regularly participates in lung cancer multidisciplinary team (MDT) work in their department. Exclusion Criteria: 1. Resident Physician Subjects: 1. Has previously participated in the construction of the GAPS evaluation set or the development of GAPS-Agent; 2. Unable to complete the tasks of the study phase. 2. Study Cases: 1. Key case information is missing, such as text-form data on pathology (including IHC/NGS), imaging, laboratory tests, prior medical history, comorbidities, or PS score; 2. Decision-making for the case is strictly dependent on non-text information. 3. Adjudication Expert Panel: 1. Participated in the construction of the GAPS evaluation set, the content validity verification, or the development of GAPS-Agent for this study; 2. Has a direct conflict of interest with any specific product among the two-arm tools of this study.