INTRODUCTION
Minimally invasive approaches have transformed colorectal surgery, offering advantages such as reduced postoperative pain, shorter hospital stays, faster recovery of bowel function, improved cosmetic outcomes, and a lower risk of complications, including infection and ileus [1]. However, laparoscopy continues to face limitations in anatomically complex areas such as the pelvis, largely due to 2-dimensional imaging, unstable camera control, and restricted instrument mobility [2].
Robotic-assisted colorectal surgery addresses many of these limitations by providing stereoscopic 3D high-definition visualization, articulated instruments, improved ergonomics, tremor filtration, motion scaling, and greater surgeon comfort [3].
Artificial intelligence (AI) has rapidly advanced within surgical practice. Techniques such as fluorescence imaging and laser speckle contrast imaging enable real-time perfusion assessment and identification of critical structures. Virtual reality enhances surgical training by providing immersive, risk-free environments with real-time feedback. In addition, AI-assisted segmentation of computed tomography (CT) or magnetic resonance imaging allows manual, semiautomated, or fully automated creation of patient-specific 3D anatomical models [4].
Despite this progress, standardizing augmented reality (AR) in colorectal surgery is particularly challenging due to the dynamic and variable anatomy of the colon. Unlike solid organs, AR models in colorectal surgery must align primarily with vascular landmarks rather than bowel contours. Moreover, superimposed 3D models can obscure surgical instruments, a phenomenon known as instrument occlusion, which creates a potentially hazardous intraoperative environment [5]. This limitation has been widely recognized as a barrier to broader AR adoption in surgery, and AI-driven de-occlusion algorithms have recently been proposed as a promising solution [6].
Here, we present the technical implementation and feasibility of the world’s first AI-assisted AR-guided robotic right hemicolectomy with real-time instrument de-occlusion, performed at the Euroclinic General Hospital (Athens, Greece).
TECHNIQUE
The patient was a 45-year-old man with a body mass index of 27 kg/m2 and no history of prior abdominal surgery or significant comorbidities. Colonoscopy and biopsy confirmed adenocarcinoma. Preoperative CT revealed a 12-cm ascending colon tumor invading the lateral abdominal wall and suspicious para-aortic lymph nodes. Triple-phase 1-mm-thick CT images were acquired for anatomical segmentation using a combination of manual and semiautomated methods (Fig. 1).
The procedure was performed using the da Vinci Xi robotic system (Intuitive Surgical) with a standard four-arm port configuration. The patient was placed in the modified Lloyd-Davis position with the operating table slightly tilted to the right, and the trocars were inserted in a reverse “smile” configuration along the right abdomen. The AI-AR integration was achieved through collaboration between the Euroclinic colorectal surgery team, AI engineers from Orsi Academy (Merelbeke-Melle, Belgium), and the hospital’s IT department.
CT image segmentation was performed using the NVIDIA Holoscan and Clara software suite (NVIDIA Corp), in collaboration with the AI-AR development team at Orsi Academy. Segmentation combined manual and semiautomated techniques applied to the triple-phase 1-mm CT datasets. The process involved the operating surgeon, radiologists, and biomedical engineers experienced in surgical AR applications.
AI-assisted instrument segmentation and de-occlusion were implemented using an NVIDIA Clara AGX developer kit (NVIDIA Corp) running a binary segmentation model (Feature Pyramid Network with EfficientNetV2 backbone), optimized with TensorRT (NVIDIA Corp) to achieve a per-frame latency of approximately 13 millisecond. Integration into the da Vinci Xi robotic platform was enabled via TilePro (Intuitive Surgical), allowing side-by-side display of the AR-enhanced feed and the native endoscopic image. The segmentation process enabled fabrication of a patient-specific 3D anatomical model, which was then integrated into the intraoperative workflow (Fig. 2).
A robotic complete mesocolic excision with D3 and para-aortic lymphadenectomy was planned. AR was used to facilitate identification of major vascular structures, including the superior mesenteric artery and superior mesenteric vein, recognition of the gastrocolic trunk of Henle, and safe peripancreatic lymph node dissection, thereby reducing the risk of iatrogenic injury.
Intraoperatively, the 3D model was manually aligned with the patient’s anatomy using vascular landmarks and superimposed on the endoscopic view. To address the problem of instrument occlusion by the AR overlay, a binary AI segmentation model was applied to detect and mask nonorganic elements, thereby maintaining clear visualization of surgical instruments (Fig. 3). The augmented AR feed was displayed to the surgeon via TilePro integration (Fig. 4). A proof-of-concept livestream was conducted using a LiveArena capture card (LiveArena Technologies) and a laptop running Microsoft Teams (Microsoft Corp).
Ethics statement
This study was approved by the Institutional Review Board of Euroclinic Hospital (No. 442024). Written informed consent was obtained from the patient for publication of the research details and clinical images. The procedure and data collection were conducted in accordance with the ethical standards of the institutional research committee and the principles of the Declaration of Helsinki.
DISCUSSION
The first documented application of AR in general surgery was reported in 2004 by a team at IRCAD/EITS (Strasbourg, France) [7]. A 45-year-old man with a 1-cm Conn adenoma of the right adrenal gland underwent laparoscopic adrenalectomy, and AR was employed to define dissection planes and accurately localize the tumor, surrounding organs, and vessels. Since then, intraoperative AR has steadily gained momentum across multiple surgical subspecialties, including urology, hepatobiliary, and colorectal surgery.
AR provides several advantages that contribute to improved surgical outcomes. It enhances spatial perception, reduces unnecessary tissue handling, and lowers the risk of iatrogenic injury. These benefits translate into shorter operative times, increased precision, and improved patient safety. Moreover, AR has significant educational value, offering realistic simulations that allow surgical trainees to practice complex procedures in a risk-free environment [8].
Our team reported the first case of robotic complete mesocolic excision in Europe using AR guidance in early 2021 [9]. The case involved a real-time 3D overlay of reconstructed vascular anatomy via the Proximie platform (Proximie), combined with remote proctoring from an Intuitive Surgical expert in London, UK. The procedure was completed successfully, with final pathology confirming a 9-cm T3N0 right colon tumor, 48 harvested lymph nodes, and an intact mesocolon. This milestone represented the beginning of a new era of AR-assisted robotic colorectal surgery, which has since advanced rapidly through the integration of AI.
Among recent innovations, the AI-powered surgical visualization system Eureka α, developed by Anaut and approved in Japan in spring 2024, represents a major step forward [10]. Equipped with precision mapping technology, the system enables intraoperative, real-time identification and overlay of connective tissue, nerves, dissection planes, vessels, and organs—offering a preview of the future of intelligent surgical navigation. Its application in colorectal surgery has already been demonstrated in robotic total mesorectal excision for rectal cancer, where AI-assisted mapping enhanced visualization of anatomical planes and critical structures [11].
Despite these advances, the challenge of instrument occlusion during robotic surgery with AR overlays has persisted. The tendency of AR models to obscure surgical tools, particularly in complex colorectal procedures, has limited their intraoperative utility. A reliable solution requires enabling the robotic platform to recognize instruments and automatically prioritize their visibility by segmenting and removing overlaid 3D content (Fig. 5).
The first in-human demonstration of real-time AI-assisted instrument de-occlusion during AR robotic surgery was reported in late 2023 by Hofman et al. [6] at Orsi Academy. Using a dataset of 31,812 images from 100 robotic partial nephrectomies, they developed a binary segmentation pipeline based on a Feature Pyramid Network with an EfficientNetV2 backbone. The system, deployed on NVIDIA Clara AGX hardware and using DELTA-12G-elp-key video input (Deltacast), achieved over 98% test accuracy. Conversion to TensorRT reduced inference time from 40.5 seconds to 5.1 seconds, ultimately reaching 13 milliseconds per-frame latency. Real-time performance was validated in 3 robotic procedures, with positive intraoperative feedback.
In our experience, the AI-assisted de-occlusion algorithm proved most valuable during phases of the operation where instrument overlap with critical anatomy occurred most frequently, particularly during vascular dissection and mesocolic plane development. These steps require precise visualization of vascular pedicles and adjacent structures, which are often obscured by robotic arms in confined working spaces. By selectively masking nonorganic elements in real time, the algorithm maintained continuous visualization of target anatomy, reduced the need for repeated camera adjustments, and improved operative flow. While beneficial throughout the procedure, the impact was most pronounced during deep dissections, where even a brief loss of visualization could increase the risk of inadvertent injury.
Our successful completion of the first live AI-assisted AR-guided robotic colorectal surgery with instrument de-occlusion represents a pivotal milestone in the adoption of advanced digital technologies in colorectal surgery. This innovation not only improves surgical precision but also sets the stage for broader integration of AR and AI into complex procedures.
Looking ahead, AI-driven personalization of surgical care has the potential to transform intraoperative decision-making. Real-time AI could analyze operative data to guide surgeons in defining optimal resection planes, identifying anatomical risk zones, or suggesting preservation strategies—enabling immediate, data-informed adjustments.
Building on this case, the next developmental stage in AI-AR instrument de-occlusion is to refine segmentation models for multi-instrument recognition, optimize performance across diverse anatomical regions, and progress toward fully automated segmentation rather than manual or semiautomated workflows. Achieving reliable, real-time automated segmentation would enhance efficiency, scalability, and reproducibility across institutions. Such technology could benefit a wide range of complex surgical scenarios where visual clarity is critical, further improving intraoperative precision and safety.
Furthermore, advances in semiautonomous or fully autonomous robotic systems may eventually enable automation of repetitive tasks such as suturing or vessel sealing, allowing the surgeon to focus on the most critical operative steps. While this remains a long-term goal, the foundation is already being laid by innovations such as the one described in this report.
CONCLUSION
This report presents the first successful clinical use of AI-assisted AR with real-time instrument de-occlusion during robotic right hemicolectomy. By integrating AI-based segmentation into the AR workflow, we overcame the longstanding limitation of instrument occlusion, thereby enhancing intraoperative visualization and surgical precision. This case suggests the technical feasibility and indicates preliminary safety of next-generation digital tools in complex colorectal surgery, establishing a foundation for broader clinical adoption of intelligent, real-time surgical navigation systems.
ARTICLE INFORMATION
-
Conflict of interest
No potential conflict of interest relevant to this article was reported.
-
Funding
None.
-
Author contributions
Conceptualization: TP, PDB, JS, AP, AM; Investigation: TP, PDB, JS, AP, AM; Methodology: TP, KE, JS; Project administration: TP; Software: PDB, JS; Supervision: AP, AM; Validation: TP, PDB, AM; Visualization: TP, JS; Writing–original draft: TP, KE; Writing–review & editing: all authors. All authors read and approved the final manuscript.
-
Additional information
This study was presented orally at the 15th Clinical Robotic Surgery Association (CRSA) Worldwide Congress on November 22, 2024, in Rome, Italy. It was also awarded Best Presentation for its clinical innovation and impact at the 9th Portsmouth Colorectal Congress (PCC) on June 12, 2025, in Hedge End, UK.
Fig. 1.Manual and semiautomated segmentation from triple-phase 1-mm computed tomography (CT) images. (A–C) Raw CT images. (D–F) Segmented views. (A, D) Axial planes. (B, E) Sagittal planes. (C, F) Coronal planes.
Fig. 2.Three-dimensional model generated from segmented computed tomography images, illustrating the right colon tumor (pink). (A) Anterior view. (B) Right lateral view. (C) Posterior view.
Fig. 3.Instrument occlusion and de-occlusion during surgery. (A) Native surgical view. (B) Three-dimensional overlay obscuring instruments. (C) Artificial intelligence–based instrument segmentation. (D) Post-segmentation de-occlusion with clear visualization.
Fig. 4.Intraoperative visualization on the robotic monitor. (A–C) Raw field. (D–F) View with 3-dimensional overlay and instrument de-occlusion.
Fig. 5.Schematic illustration of the instrument occlusion problem and de-occlusion solution during augmented reality (AR)-guided robotic surgery. (A) Native intraoperative view during robotic right hemicolectomy. (B) Obstruction of surgical instruments by the 3-dimensional AR overlay (tumor in green, vessels in red). (C) Instrument detection and segmentation. (D) Post-segmentation de-occlusion for clear visualization.
REFERENCES
- 1. Chen E, Chen L, Zhang W. Robotic-assisted colorectal surgery in colorectal cancer management: a narrative review of clinical efficacy and multidisciplinary integration. Front Oncol 2025;15:1502014.ArticlePubMedPMC
- 2. Petropoulou T, Theodoraki K, Kitsanta P, Amin S. Efficiency of the robotic platform in improving the rate of sphincter preservation in patients with mid and low rectal cancer. World J Oncol 2023;14:499–504. ArticlePubMedPMC
- 3. Petropoulou T, Mpakas A, Gallou G, Tataridi O, Polydorou A. Robotic excision of anal pouch in a patient with multiple previous abdominal laparotomies: is it feasible? A video vignette. Colorectal Dis 2024;26:2007–8. ArticlePubMed
- 4. 31st International Congress of the European Association for Endoscopic Surgery (EAES) Rome, Italy, 20-23 June 2023. Surg Endosc 2024;38(Suppl 2):433–817. ArticlePDF
- 5. Fernando Trebolle J, Perez Serrano N, Yanez Benitez C, Menal Munoz P, Luesma Bartolome MJ, Cantarero Carmona I, et al. Augmented reality technology applied to surgical planning in oncological colorectal surgery. Clin Radiol Imaging J 2023;7:000204.Article
- 6. Hofman J, De Backer P, Manghi I, Simoens J, De Groote R, Van Den Bossche H, et al. First-in-human real-time AI-assisted instrument deocclusion during augmented reality robotic surgery. Healthc Technol Lett 2024;11:33–9. ArticlePubMedPMC
- 7. Marescaux J, Rubino F, Arenas M, Mutter D, Soler L. Augmented-reality-assisted laparoscopic adrenalectomy. JAMA 2004;292:2214–5. Article
- 8. Brockmeyer P, Wiechens B, Schliephake H. The role of augmented reality in the advancement of minimally invasive surgery procedures: a scoping review. Bioengineering (Basel) 2023;10:501.ArticlePubMedPMC
- 9. Petropoulou T, Polydorou A, Amin S. First robotic CME in Europe with augmented reality tools. Tech Coloproctol 2021;25:887–8. ArticlePubMedPDF
- 10. Biospace. Anaut announces Japanese regulatory approval of AI-powered surgical visualization tool, Eureka α [Internet]. BioSpace; 2024 [cited 2025 Jun 28]. Available from: https://www.biospace.com/anaut-announces-japanese-regulatory-approval-of-ai-powered-surgical-visualization-tool-eureka-α
- 11. Piozzi GN, Duhoky R, Przedlacka A, Ronconi Di Giuseppe D, Khan JS. Artificial intelligence real-time mapping with Eureka during robotic total mesorectal excision for rectal cancer: a video vignette. Colorectal Dis 2025;27:e17240. ArticlePubMed
Citations
Citations to this article as recorded by
- Robotic-Assisted Resection of a Giant Mesenteric Lymphangioma in a 36-Year-Old Woman: A Case Report
Thalia Petropoulou, Kyriacos Evangelou, Andreas Polydorou
Cureus.2026;[Epub] CrossRef
KSCP
APFCP
SIUCP-ETS
NSCRD
SCRS
, Pieter De Backer3
Cite this Article
XML Download
