RS23 - Large Language Models Routinely Overcode Peripheral Endovascular Procedures Relative to Professional Coders

Objectives: Large language models (LLMs) have demonstrated an aptitude for a wide range of back-end and administrative tasks relevant to supporting a vascular surgery practice. Accurate reporting of Current Procedural Terminology (CPT) codes is critical for reimbursement, compliance, and quality reporting. This is particularly true for outpatient endovascular procedures, where coding is frequently complicated by bundling, changing definitions, and site-of-service requirements. This study sought to examine the accuracy of LLM generated codes in comparison to those generated by professional medical coders.

Methods: 110 consecutive peripheral endovascular procedures performed by three vascular surgeons at a tertiary care community hospital Cardiovascular Lab were retrieved. The text of these reports, excluding any identifying information or summarizations of findings, were provided to a large language model (Chat GPT 5.1, OpenAI, free version, with “use data to help model” mode disabled). The following prompt was used to generate CPT codes for these operative reports: “You work in the medical billing and coding department of a vascular surgery practice. The following operative report was submitted, and your goal is to submit an accurate list of 2025 CPT codes, with applicable modifiers, for billing purposes. Only report the final line items in the physician professional claim for each operative report.” Moderate sedation codes were excluded as they were inconsistently reported in the operative reports. CPT codes generated by the LLM were then compared to those submitted by the hospital’s coding and billing department.

Results: Of the 110 operative reports, one case resulted in an exact match for codes generated by the LLM and codes submitted by the professional coders. On average the LLM generated 5.3 CPT codes per report vs 3.8 from the professional coders. Based on the text of the operative reports, the LLM generated excess codes 71% of the time, the same number of codes 19% of the time, and fewer codes in 10% of cases.

Conclusions: Use of a contemporary, mass-market large language model to generate CPT codes from operative reports alone was associated with poor overall accuracy and a consistent tendency toward overcoding. These findings highlight important limitations in the current applicability of LLMs for autonomous procedural billing. Future models may achieve improved performance through targeted training on specialty-specific operative reports and formal incorporation of CPT coding rules, bundling conventions, and payer requirements, potentially enabling their use as adjunctive, rather than standalone, tools in surgical coding workflows.