SS09 - ReflectOR: an LLM-based Agent for Post-Operative Surgical Debriefing

Objectives: Ineffective teamwork and communication are major contributors to medical errors in surgical environments, making post-operative debriefings critical for improving team performance and patient safety. However, debriefings are often rushed, incomplete, or reliant on memory due to clinicians’ time constraints. Recent advances in Large Language Models (LLMs) offer new opportunities to support structured, data-driven debriefings, yet most existing solutions are limited to English and single-task pipelines.
This study introduces ReflectOR, an agentic AI system designed to support post-operative surgical debriefing by automatically processing intraoperative audio recordings. The objectives were: (1) to evaluate whether current transcription and speaker diarization models can achieve satisfactory performance on Italian surgical audio in a zero-shot setting, and (2) to assess the feasibility and perceived utility of an LLM-based multi-agent architecture for surgical debriefings.

Methods: ReflectOR integrates an audio-processing pipeline (Figure 1) with an agentic AI architecture composed of a central coordinator and specialized LLM-based sub-agents (Figure 2). Audio from five simulated endovascular aneurysm repair (EVAR) procedures, performed by vascular surgeons using Surgical Science Angiomentor, were preprocessed for noise reduction, then automatically transcribed and diarized using state-of-the-art models, including Whisper, NVIDIA NeMo, Pyannote-Audio, and Google Gemini. Performance was quantitatively evaluated against manually annotated ground truth using Word Error Rate (WER), defined as the percentage of substituted, inserted, or deleted words required to align the automatic transcription with the manual reference, and Diarization Error Rate (DER), defined as the percentage of utterances assigned to an incorrect speaker. The agentic system was further evaluated through structured feedback from two vascular surgeons.

Results: Among the evaluated models, GEMINI-2.5-PRO achieved the best performance, with a WER of 11% and a DER of 11%, outperforming both open-source and alternative commercial solutions in a zero-shot setting. Surgeon evaluation highlighted strengths including support for error analysis, quantification of improvements across simulations, discussion of team roles to enhance procedural fluency, efficiency and precision, provision of cost-related information, and a concise patient summary. Limitations included suboptimal event timeline reconstruction, partial redundancy between summaries and discussion plans, and incomplete material tracking affecting cost estimation.

Conclusions: ReflectOR demonstrates the potential of agentic LLM-based systems to support post-operative surgical debriefing, even in non-English clinical settings. While promising, real-world deployment will require improved audio capture, validation on larger datasets, and careful consideration of ethical, privacy, and workflow-integration challenges.