Author name cluster

Franziska Mathis-Ullrich

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

12 papers

2 author rows

AAAI Conference 2026 Conference Paper

LUDO: Low-Latency Understanding of Deformable Objects Using Point Cloud Occupancy Functions (Abstract Reprint)

Pit Henrich
Franziska Mathis-Ullrich
Paul Maria Scheikl

Accurately determining the shape of deformable objects and the location of their internal structures is crucial for medical tasks that require precise targeting, such as robotic biopsies. We introduce LUDO, a method for accurate low-latency understanding of deformable objects. LUDO reconstructs objects in their deformed state, including their internal structures, from a single-view point cloud observation in under 30 ms using occupancy networks. LUDO provides uncertainty estimates for its predictions. Additionally, it provides explainability by highlighting key features in its input observations. Both uncertainty and explainability are important for safety-critical applications such as surgery. We evaluate LUDO in real-world robotic experiments, achieving a success rate of 98.9% for puncturing various regions of interest (ROIs) inside deformable objects. We compare LUDO to a popular baseline and show its superior ROI localization accuracy, training time, and memory requirements. LUDO demonstrates the potential to interact with deformable objects without the need for deformable registration methods.

PDF Details DOI

IROS Conference 2025 Conference Paper

Designing a Magnetic Endoscope for In Vivo Contact-Based Tissue Scanning Using Developable Roller

Nikita J. Greenidge
Christian Marzi
Benjamin Calmé
James W. Martin
Bruno Scaglioni
Franziska Mathis-Ullrich
Pietro Valdastri

Magnetic manipulation has been adopted as a method of actuation in both wireless capsule endoscopy and soft-tethered endoscopy, with the goal of improving gastrointestinal procedures. However, by nature of magnetic manipulation, these endoscopes are typically limited to a maximum of five degrees of freedom (DoF). With the need to introduce additional contact-based sensing modalities for subsurface investigation into these systems as well as to improve overall dexterity, it is both practically and clinically beneficial to recover the lost DoF i. e. the roll around the main axis. This paper presents a method of achieving the magnetic manipulation of an underactuated device by leveraging developable surfaces, specifically, the oloid shape. The design of a clinically relevant magnetic endoscope with all its ancillary elements, as well as contact sensors, is proposed and demonstrated in vivo. The contact sensor data from the in vivo experiments show that for sweeping motions over 100° of roll, contact between the endoscope’s sensor region and the colon wall can be maintained for 74% of the motion.

Details

ICRA Conference 2024 Conference Paper

Lens Capsule Tearing in Cataract Surgery using Reinforcement Learning

Rebekka Charlotte Peter
Steffen Peikert
Ludwig Haide
Doan Xuan Viet Pham
Tahar Chettaoui
Eleonora Tagliabue
Paul Maria Scheikl
Johannes Fauser

Cataract is the leading cause of blindness worldwide with an increasing number of patients due to changing demographics, making automation an important part in future surgical treatment. In this work, we focus on a substep of cataract surgery, the Continuous Curvilinear Capsulorhexis (CCC). With a high complexity, this task is an ideal candidate for Reinforcement Learning (RL) in simulation. First, we present an interactive and physically realistic simulation based on the Finite Element Method (FEM) that mimics the tearing behavior of soft tissue during CCC. Then, we train and evaluate RL models in simulation, demonstrating that the trained policies can complete the CCC in 85% of cases. We also show that applying domain randomization techniques make the policy more robust against changes in geometrical and biomechanical boundary conditions.

Details

IROS Conference 2024 Conference Paper

Tracking Tumors under Deformation from Partial Point Clouds using Occupancy Networks

Pit Henrich
Jiawei Liu
Jiawei Ge 0001
Samuel Schmidgall
Lauren M. Shepard
Ahmed Ezzat Ghazi
Franziska Mathis-Ullrich
Axel Krieger

To track tumors during surgery, information from preoperative CT scans is used to determine their position. However, as the surgeon operates, the tumor may be deformed which presents a major hurdle for accurately resecting the tumor, and can lead to surgical inaccuracy, increased operation time, and excessive margins. This issue is particularly pronounced in robot-assisted partial nephrectomy (RAPN), where the kidney undergoes significant deformations during operation. Toward addressing this, we introduce a occupancy network-based method for the localization of tumors within kidney phantoms undergoing deformations at interactive speeds. We validate our method by introducing a 3D hydrogel kidney phantom embedded with exophytic and endophytic renal tumors. It closely mimics real tissue mechanics to simulate kidney deformation during in vivo surgery, providing excellent contrast and clear delineation of tumor margins to enable automatic threshold-based segmentation. Our findings indicate that the proposed method can localize tumors in moderately deforming kidneys with a margin of 6mm to 10mm, while providing essential volumetric 3D information at over 60Hz. This capability directly enables downstream tasks such as robotic resection.

Details

ICLR Conference 2023 Conference Paper

Grounding Graph Network Simulators using Physical Sensor Observations

Jonas Linkerhägner
Niklas Freymuth
Paul Maria Scheikl
Franziska Mathis-Ullrich
Gerhard Neumann

Physical simulations that accurately model reality are crucial for many engineering disciplines such as mechanical engineering and robotic motion planning. In recent years, learned Graph Network Simulators produced accurate mesh-based simulations while requiring only a fraction of the computational cost of traditional simulators. Yet, the resulting predictors are confined to learning from data generated by existing mesh-based simulators and thus cannot include real world sensory information such as point cloud data. As these predictors have to simulate complex physical systems from only an initial state, they exhibit a high error accumulation for long-term predictions. In this work, we integrate sensory information to ground Graph Network Simulators on real world observations. In particular, we predict the mesh state of deformable objects by utilizing point cloud data. The resulting model allows for accurate predictions over longer time horizons, even under uncertainties in the simulation, such as unknown material properties. Since point clouds are usually not available for every time step, especially in online settings, we employ an imputation-based model. The model can make use of such additional information only when provided, and resorts to a standard Graph Network Simulator, otherwise. We experimentally validate our approach on a suite of prediction tasks for mesh-based interactions between soft and rigid bodies. Our method results in utilization of additional point cloud information to accurately predict stable simulations where existing Graph Network Simulators fail.

Details

JMLR Journal 2023 Journal Article

LapGym - An Open Source Framework for Reinforcement Learning in Robot-Assisted Laparoscopic Surgery

Paul Maria Scheikl
Balázs Gyenes
Rayan Younis
Christoph Haas
Gerhard Neumann
Martin Wagner
Franziska Mathis-Ullrich

Recent advances in reinforcement learning (RL) have increased the promise of introducing cognitive assistance and automation to robot-assisted laparoscopic surgery (RALS). However, progress in algorithms and methods depends on the availability of standardized learning environments that represent skills relevant to RALS. We present LapGym, a framework for building RL environments for RALS that models the challenges posed by surgical tasks, and sofaenv, a diverse suite of 12 environments. Motivated by surgical training, these environments are organized into 4 tracks: Spatial Reasoning, Deformable Object Manipulation & Grasping, Dissection, and Thread Manipulation. Each environment is highly parametrizable for increasing difficulty, resulting in a high performance ceiling for new algorithms. We use Proximal Policy Optimization (PPO) to establish a baseline for model-free RL algorithms, investigating the effect of several environment parameters on task difficulty. Finally, we show that many environments and parameter configurations reflect well-known, open problems in RL research, allowing researchers to continue exploring these fundamental problems in a surgical context. We aim to provide a challenging, standard environment suite for further development of RL for RALS, ultimately helping to realize the full potential of cognitive surgical robotics. LapGym is publicly accessible through GitHub (https://github.com/ScheiklP/lap_gym). [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2023. ( edit, beta )

PDF Details

ICRA Conference 2022 Conference Paper

Automated Linear and Non-Linear Path Planning for Neurosurgical Interventions

Steffen Peikert
Christian Kunz
Nikola Fischer
Michal Hlavác
Andrej Pala
Max Schneider
Franziska Mathis-Ullrich

Recent advances in medical technology have produced a number of flexible instruments that are capable of traversing non-linear paths. This is of special interest in the field of neurosurgery. However, the non-rigid instruments have the disadvantage that path planning becomes increasingly difficult. In addition to anatomical risk factors, the mechanical properties and constraints of the specific instrument must also be considered. To support surgeons to deal with the increase in planning complexity, we present a novel method for both linear and arbitrary follow-the-leader flexible path planning. Our method is utilizing patient-specific image data, which is then used to generate a multi-objective problem consisting of conventional risk metrics for path planning in high-risk regions like the accumulated path cost or the distance to risk structures. Simultaneously, the path-problem is also constraint to mechanical properties of the instrument such as curvature or maximum operational length. Optimal paths can then be generated by solving a multi-objective problem by approximating the Pareto front. We show that our method can automatically generate linear and non-linear paths for neurosurgical interventions in the human brain in less than 2 minutes. Furthermore, we show that the proposed automated method generates paths with 87% reduced risk compared to standard of care plannings.

Details

ICRA Conference 2022 Conference Paper

Capacitive Proximity Sensor for Non-Contact Endoscope Localization

Christian Marzi
Hosam Alagi
Olivia Rau
Jochen Hampe
Jan G. Korvink
Björn Hein
Franziska Mathis-Ullrich

The promising automation of flexible surgical instruments and robots is impeded by the lack of sensory means, which allow for sensing of an instrument's position to the surrounding tissue. This work presents a novel sensory method utilizing capacitive proximity sensing to derive a relative localization of a flexible instrument inside a hollow organ. The method is evaluated by exemplary integration of a sensor in a commercial gastroendoscope and accuracy analysis using a high precision robot. The results show an accuracy of distance sensing from a medical phantom's center of 2%. The method is also evaluated for the irregularly shaped surrounding of ex-vivo tissue in a dynamic scenario. This promising approach holds potential for transfer to clinical scenarios and for further development towards pose estimation of flexible surgical robots and shape sensing of a minimally invasive environment.

Details

IROS Conference 2022 Conference Paper

LapSeg3D: Weakly Supervised Semantic Segmentation of Point Clouds Representing Laparoscopic Scenes

Benjamin Alt
Christian Kunz
Darko Katic
Rayan Younis
Rainer Jäkel
Beat Peter Müller-Stich
Martin Wagner 0001
Franziska Mathis-Ullrich

The semantic segmentation of surgical scenes is a prerequisite for task automation in robot assisted interventions. We propose LapSeg3D, a novel DNN-based approach for the voxel-wise annotation of point clouds representing surgical scenes. As the manual annotation of training data is highly time consuming, we introduce a semi-autonomous clustering-based pipeline for the annotation of the gallbladder, which is used to generate segmented labels for the DNN. When evaluated against manually annotated data, LapSeg3D achieves an F1 score of 0. 94 for gallbladder segmentation on various datasets of ex-vivo porcine livers. We show LapSeg3D to generalize accurately across different gallbladders and datasets recorded with different RGB-D camera systems.

Details

IROS Conference 2021 Conference Paper

Cooperative Assistance in Robotic Surgery through Multi-Agent Reinforcement Learning

Paul Maria Scheikl
Balázs Gyenes
Tornike Davitashvili
Rayan Younis
André Schulze
Beat Peter Müller-Stich
Gerhard Neumann
Martin Wagner 0001

Cognitive cooperative assistance in robot-assisted surgery holds the potential to increase quality of care in minimally invasive interventions. Automation of surgical tasks promises to reduce the mental exertion and fatigue of surgeons. In this work, multi-agent reinforcement learning is demonstrated to be robust to the distribution shift introduced by pairing a learned policy with a human team member. Multi-agent policies are trained directly from images in simulation to control multiple instruments in a sub task of the minimally invasive removal of the gallbladder. These agents are evaluated individually and in cooperation with humans to demonstrate their suitability as autonomous assistants. Compared to human teams, the hybrid teams with artificial agents perform better considering completion time (44. 4% to 71. 2% shorter) as well as number of collisions (44. 7% to 98. 0% fewer). Path lengths, however, increase under control of an artificial agent (11. 4% to 33. 5% longer). A multi-agent formulation of the learning problem was favored over a single-agent formulation on this surgical sub task, due to the sequential learning of the two instruments. This approach may be extended to other tasks that are difficult to formulate within the standard reinforcement learning framework. Multi-agent reinforcement learning may shift the paradigm of cognitive robotic surgery towards seamless cooperation between surgeons and assistive technologies.

Details

ICRA Conference 2015 Conference Paper

Magnetically actuated and guided milli-gripper for medical applications

Franziska Mathis-Ullrich
Kanika S. Dheman
Simone Schürle
Bradley J. Nelson

This paper presents the design, kinematics, fabrication, and magnetic manipulation of a milli-gripper for medical applications. The design employs a permanent magnet for two purposes. It actuates the compliant gripper and allows for maneuverability of the milli-gripper in an externally applied magnetic field generated by an electromagnetic manipulation system. The modular milli-gripper can be manipulated directly or attached to the distal tip of a magnetically steered catheter. Experiments show successful actuation of the gripper and guidance of the device with the integrated gripper in both the tethered and untethered configuration.

Details

ICRA Conference 2014 Conference Paper

Automated capsulorhexis based on a hybrid magnetic-mechanical actuation system

Franziska Mathis-Ullrich
Simone Schürle
Roel Pieters
Avraham Dishy
Stephan Michels
Bradley J. Nelson

This paper presents a hybrid magnetic-mechanical manipulation system for automated capsulorhexis utilizing a flexible catheter with a sharp edge magnetic tip. Vision based closed loop control is implemented to guide the tip on a circular path in the anterior eye segment. A continuous motion with high repeatability is achieved. The system shows the first catheter-based application of the electromagnetic manipulation system, OctoMag, for fast and safe ophthalmic surgery that potentially reduces the risk of complications and improves precision.

Details