Author name cluster

Paul Maria Scheikl

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

2 author rows

AAAI Conference 2026 Conference Paper

LUDO: Low-Latency Understanding of Deformable Objects Using Point Cloud Occupancy Functions (Abstract Reprint)

Pit Henrich
Franziska Mathis-Ullrich
Paul Maria Scheikl

Accurately determining the shape of deformable objects and the location of their internal structures is crucial for medical tasks that require precise targeting, such as robotic biopsies. We introduce LUDO, a method for accurate low-latency understanding of deformable objects. LUDO reconstructs objects in their deformed state, including their internal structures, from a single-view point cloud observation in under 30 ms using occupancy networks. LUDO provides uncertainty estimates for its predictions. Additionally, it provides explainability by highlighting key features in its input observations. Both uncertainty and explainability are important for safety-critical applications such as surgery. We evaluate LUDO in real-world robotic experiments, achieving a success rate of 98.9% for puncturing various regions of interest (ROIs) inside deformable objects. We compare LUDO to a popular baseline and show its superior ROI localization accuracy, training time, and memory requirements. LUDO demonstrates the potential to interact with deformable objects without the need for deformable registration methods.

PDF Details DOI

IROS Conference 2025 Conference Paper

From Monocular Vision to Autonomous Action: Guiding Tumor Resection via 3D Reconstruction

Ayberk Acar
Mariana E. Smith
Lidia Al-Zogbi
Tanner Watts
Fangjie Li
Hao Li 0108
Nural Yilmaz
Paul Maria Scheikl

Surgical automation requires precise guidance and understanding of the scene. Current methods in the literature rely on bulky depth cameras to create maps of the anatomy; however, this does not translate well to space-limited clinical applications. Monocular cameras are small and allow minimally invasive surgeries in tight spaces, but additional processing is required to generate 3D scene understanding. We propose a 3D mapping pipeline that uses only RGB images to create segmented point clouds of the target anatomy. To ensure the most accurate reconstruction, we compare different structure from motion algorithms’ performance on mapping the central airway obstructions, and test the pipeline on a downstream task of tumor resection. In several metrics, including post-procedure percentage tissue charring, our pipeline performs comparably to RGB-D cameras and, in some cases, even surpasses their downstream task performance. These promising results demonstrate that automation guidance can be achieved in minimally invasive procedures with monocular cameras. This study is a step toward the complete autonomy of surgical robots.

Details

IROS Conference 2025 Conference Paper

SurgiPose: Estimating Surgical Tool Kinematics from Monocular Video for Surgical Robot Learning

Juo-Tung Chen
Xinhao Chen
Ji Woong Kim
Paul Maria Scheikl
Richard Jaepyeong Cha
Axel Krieger

Imitation learning (IL) has shown immense promise in enabling autonomous dexterous manipulations, including in learning surgical tasks. To fully unlock the potential of IL for surgery, access to clinical datasets is needed, which unfortunately lack the kinematic data required for current IL approaches. A promising source of large-scale surgical demonstrations is monocular surgical videos available online, making monocular pose estimation a crucial step toward enabling large-scale robot learning. Towards this end, we propose SurgiPose, a differentiable rendering-based approach to estimate kinematic information from monocular surgical videos, eliminating the need for direct access to ground-truth kinematics. Our method infers tool trajectories and joint angles by optimizing tool pose parameters to minimize the discrepancy between rendered and real images. To evaluate the effectiveness of our approach, we conduct experiments on two robotic surgical tasks—tissue lifting and needle pickup—using the da Vinci Research Kit Si (dVRK Si). We train imitation learning policies with both ground-truth measured kinematics and with estimated kinematics from video and compare their performance. Our results show that policies trained on estimated kinematics achieve comparable success rates to those trained on ground-truth data, demonstrating the feasibility of using monocular video-based kinematic estimation for surgical robot learning. By enabling kinematic estimation from monocular surgical videos, our work lays the foundation for large-scale learning of autonomous surgical policies from online surgical data.

Details

ICRA Conference 2024 Conference Paper

Lens Capsule Tearing in Cataract Surgery using Reinforcement Learning

Rebekka Charlotte Peter
Steffen Peikert
Ludwig Haide
Doan Xuan Viet Pham
Tahar Chettaoui
Eleonora Tagliabue
Paul Maria Scheikl
Johannes Fauser

Cataract is the leading cause of blindness worldwide with an increasing number of patients due to changing demographics, making automation an important part in future surgical treatment. In this work, we focus on a substep of cataract surgery, the Continuous Curvilinear Capsulorhexis (CCC). With a high complexity, this task is an ideal candidate for Reinforcement Learning (RL) in simulation. First, we present an interactive and physically realistic simulation based on the Finite Element Method (FEM) that mimics the tearing behavior of soft tissue during CCC. Then, we train and evaluate RL models in simulation, demonstrating that the trained policies can complete the CCC in 85% of cases. We also show that applying domain randomization techniques make the policy more robust against changes in geometrical and biomechanical boundary conditions.

Details

ICLR Conference 2023 Conference Paper

Grounding Graph Network Simulators using Physical Sensor Observations

Jonas Linkerhägner
Niklas Freymuth
Paul Maria Scheikl
Franziska Mathis-Ullrich
Gerhard Neumann

Physical simulations that accurately model reality are crucial for many engineering disciplines such as mechanical engineering and robotic motion planning. In recent years, learned Graph Network Simulators produced accurate mesh-based simulations while requiring only a fraction of the computational cost of traditional simulators. Yet, the resulting predictors are confined to learning from data generated by existing mesh-based simulators and thus cannot include real world sensory information such as point cloud data. As these predictors have to simulate complex physical systems from only an initial state, they exhibit a high error accumulation for long-term predictions. In this work, we integrate sensory information to ground Graph Network Simulators on real world observations. In particular, we predict the mesh state of deformable objects by utilizing point cloud data. The resulting model allows for accurate predictions over longer time horizons, even under uncertainties in the simulation, such as unknown material properties. Since point clouds are usually not available for every time step, especially in online settings, we employ an imputation-based model. The model can make use of such additional information only when provided, and resorts to a standard Graph Network Simulator, otherwise. We experimentally validate our approach on a suite of prediction tasks for mesh-based interactions between soft and rigid bodies. Our method results in utilization of additional point cloud information to accurately predict stable simulations where existing Graph Network Simulators fail.

Details

JMLR Journal 2023 Journal Article

LapGym - An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Paul Maria Scheikl
Balázs Gyenes
Rayan Younis
Christoph Haas
Gerhard Neumann
Martin Wagner
Franziska Mathis-Ullrich

Recent advances in reinforcement learning (RL) have increased the promise of introducing cognitive assistance and automation to robot-assisted laparoscopic surgery (RALS). However, progress in algorithms and methods depends on the availability of standardized learning environments that represent skills relevant to RALS. We present LapGym, a framework for building RL environments for RALS that models the challenges posed by surgical tasks, and sofaenv, a diverse suite of 12 environments. Motivated by surgical training, these environments are organized into 4 tracks: Spatial Reasoning, Deformable Object Manipulation & Grasping, Dissection, and Thread Manipulation. Each environment is highly parametrizable for increasing difficulty, resulting in a high performance ceiling for new algorithms. We use Proximal Policy Optimization (PPO) to establish a baseline for model-free RL algorithms, investigating the effect of several environment parameters on task difficulty. Finally, we show that many environments and parameter configurations reflect well-known, open problems in RL research, allowing researchers to continue exploring these fundamental problems in a surgical context. We aim to provide a challenging, standard environment suite for further development of RL for RALS, ultimately helping to realize the full potential of cognitive surgical robotics. LapGym is publicly accessible through GitHub (https://github.com/ScheiklP/lap_gym). [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2023. ( edit, beta )

PDF Details

IROS Conference 2021 Conference Paper

Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning

Paul Maria Scheikl
Balázs Gyenes
Tornike Davitashvili
Rayan Younis
André Schulze
Beat Peter Müller-Stich
Gerhard Neumann
Martin Wagner 0001

Cognitive cooperative assistance in robot-assisted surgery holds the potential to increase quality of care in minimally invasive interventions. Automation of surgical tasks promises to reduce the mental exertion and fatigue of surgeons. In this work, multi-agent reinforcement learning is demonstrated to be robust to the distribution shift introduced by pairing a learned policy with a human team member. Multi-agent policies are trained directly from images in simulation to control multiple instruments in a sub task of the minimally invasive removal of the gallbladder. These agents are evaluated individually and in cooperation with humans to demonstrate their suitability as autonomous assistants. Compared to human teams, the hybrid teams with artificial agents perform better considering completion time (44. 4% to 71. 2% shorter) as well as number of collisions (44. 7% to 98. 0% fewer). Path lengths, however, increase under control of an artificial agent (11. 4% to 33. 5% longer). A multi-agent formulation of the learning problem was favored over a single-agent formulation on this surgical sub task, due to the sequential learning of the two instruments. This approach may be extended to other tasks that are difficult to formulate within the standard reinforcement learning framework. Multi-agent reinforcement learning may shift the paradigm of cognitive robotic surgery towards seamless cooperation between surgeons and assistive technologies.

Details