Arrow Research search

Author name cluster

Jens Behley

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

41 papers
2 author rows

Possible papers

41

AAAI Conference 2026 Conference Paper

Radar Instance Transformer: Reliable Moving Instance Segmentation in Sparse Radar Point Clouds (Abstract Reprint)

  • Matthias Zeller
  • Vardeep Singh Sandhu
  • Benedikt Mersch
  • Jens Behley
  • Michael Heidingsfeld
  • Cyrill Stachniss

The perception of moving objects is crucial for autonomous robots performing collision avoidance in dynamic environments. LiDARs and cameras tremendously enhance scene interpretation but do not provide direct motion information and face limitations under adverse weather. Radar sensors overcome these limitations and provide Doppler velocities, delivering direct information on dynamic objects. In this article, we address the problem of moving instance segmentation in radar point clouds to enhance scene interpretation for safety-critical tasks. Our radar instance transformer enriches the current radar scan with temporal information without passing aggregated scans through a neural network. We propose a full-resolution backbone to prevent information loss in sparse point cloud processing. Our instance transformer head incorporates essential information to enhance segmentation but also enables reliable, class-agnostic instance assignments. In sum, our approach shows superior performance on the new moving instance segmentation benchmarks, including diverse environments, and provides model-agnostic modules to enhance scene interpretation.

PDF Details DOI

IROS Conference 2025 Conference Paper

3D Hierarchical Panoptic Segmentation in Real Orchard Environments Across Different Sensors

  • Matteo Sodano
  • Federico Magistri
  • Elias Marks
  • Fares Hosn
  • Aibek Zurbayev
  • Rodrigo Marcuzzi
  • Meher V. R. Malladi
  • Jens Behley

Crop yield estimation is a relevant problem in agriculture, because an accurate yield estimate can support farmers’ decisions on harvesting or precision intervention. Robots can help to automate this process. To do so, they need to be able to perceive the surrounding environment to identify target objects such as trees and plants. In this paper, we introduce a novel approach to address the problem of hierarchical panoptic segmentation of apple orchards on 3D data from different sensors. Our approach is able to simultaneously provide semantic segmentation, instance segmentation of trunks and fruits, and instance segmentation of trees (a trunk with its fruits). This allows us to identify relevant information such as individual plants, fruits, and trunks, and capture the relationship among them, such as precisely estimate the number of fruits associated to each tree in an orchard. To efficiently evaluate our approach for hierarchical panoptic segmentation, we provide a dataset designed specifically for this task. Our dataset is recorded in Bonn, Germany, in a real apple orchard with a variety of sensors, spanning from a terrestrial laser scanner to a RGB-D camera mounted on different robots platforms. The experiments show that our approach surpasses state-of-the-art approaches in 3D panoptic segmentation in the agricultural domain, while also providing full hierarchical panoptic segmentation. Our dataset is publicly available at https://www.ipb.uni-bonn.de/data/hops/.The open-source implementation of our approach is available at https://github.com/PRBonn/hapt3D.

Details

ICRA Conference 2025 Conference Paper

A Dataset and Benchmark for Shape Completion of Fruits for Agricultural Robotics

  • Federico Magistri
  • Thomas Läbe
  • Elias Marks
  • Sumanth Nagulavancha
  • Yue Pan 0009
  • Claus Smitt
  • Lasse Klingbeil
  • Michael Halstead

As the world population is expected to reach 10 billion by 2050, our agricultural production system needs to double its productivity despite a decline of human workforce in the agricultural sector. Autonomous robotic systems are one promising pathway to increase productivity by taking over labor-intensive manual tasks like fruit picking. To be effective, such systems need to monitor and interact with plants and fruits precisely, which is challenging due to the cluttered nature of agricultural environments causing, for example, strong occlusions. Thus, being able to estimate the complete 3D shapes of objects in presence of occlusions is crucial for automating operations such as fruit harvesting. In this paper, we propose the first publicly available 3D shape completion dataset for agricultural vision systems. We provide an RGB-D dataset for estimating the 3D shape of fruits. Specifically, our dataset contains RGB-D frames of single sweet peppers in lab conditions but also in a commercial greenhouse. For each fruit, we additionally collected high-precision point clouds that we use as ground truth. For acquiring the ground truth shape, we developed a measuring process that allows us to record data of real sweet pepper plants, both in the lab and in the greenhouse with high precision, and determine the shape of the sensed fruits. We release our dataset, consisting of almost 7, 000 RGB-D frames belonging to more than 100 different fruits. We provide segmented RGB-D frames, with camera intrinsics to easily obtain colored point clouds, together with the corresponding high-precision, occlusion-free point clouds obtained with a high-precision laser scanner. We additionally enable evaluation of shape completion approaches on a hidden test set through a public challenge on a benchmark server.

Details

IROS Conference 2025 Conference Paper

Benchmark for Evaluating Long-Term Localization in Indoor Environments under Substantial Static and Dynamic Scene Changes

  • Niklas Trekel
  • Tiziano Guadagnino
  • Thomas Läbe
  • Louis Wiesmann
  • Perrine Aguiar
  • Jens Behley
  • Cyrill Stachniss

Accurate localization is crucial for the autonomous operation of mobile robots. Specifically for indoor scenarios, localization algorithms typically rely on a previously generated map. However, many real-world sites like warehouses or healthcare environments violate the underlying assumption that the robot’s surroundings are mainly static. In this paper, we introduce a new dataset plus a benchmark that enables evaluating and comparing indoor localization methods in complex and changing real-world scenarios. While several datasets for indoor scenes exist, only a few combine the long-term localization aspect of repeatedly revisiting the same environment under varying conditions with precise ground truth over multiple rooms. Our dataset comprises various sequences recorded with a wheeled robot covering an office environment. We provide data from two 2D LiDARs, multiple consumer-grade RGB-D cameras, and the robot’s wheel odometry. By densely placing fiducial markers on every room ceiling, we can also provide accurate pose information within a single global frame for the whole environment, estimated through an additional upward-facing camera. We evaluate existing localization algorithms on our data and make the dataset together with a server-based benchmark evaluation publicly available. This facilitates an unbiased evaluation of localization approaches and enables further research on their application in challenging indoor scenarios.

Details

IROS Conference 2025 Conference Paper

Coherent Online Road Topology Estimation and Reasoning with Standard-Definition Maps

  • Khanh Son Pham
  • Christian Witte
  • Jens Behley
  • Johannes Betz
  • Cyrill Stachniss

Most autonomous cars rely on the availability of high-definition (HD) maps. Current research aims to address this constraint by directly predicting HD map elements from onboard sensors and reasoning about the relationships between the predicted map and traffic elements. Despite recent advancements, the coherent online construction of HD maps remains a challenging endeavor, as it necessitates modeling the high complexity of road topologies in a unified and consistent manner. To address this challenge, we propose a coherent approach to predict lane segments and their corresponding topology, as well as road boundaries, all by leveraging prior map information represented by commonly available standard-definition (SD) maps. We propose a network architecture, which leverages hybrid lane segment encodings comprising prior information and denoising techniques to enhance training stability and performance. Furthermore, we facilitate past frames for temporal consistency. Our experimental evaluation demonstrates that our approach outperforms previous methods by a large margin, highlighting the benefits of our modeling scheme.

Details

ICRA Conference 2025 Conference Paper

Digiforests: a Longitudinal Lidar Dataset for Forestry Robotics

  • Meher V. R. Malladi
  • Nived Chebrolu
  • Irene Scacchetti
  • Luca Lobefaro
  • Tiziano Guadagnino
  • Benoît Casseau
  • Haedam Oh
  • Leonard Freißmuth

Forests are vital to our ecosystems, acting as carbon sinks, climate stabilizers, biodiversity centers, and wood sources. Due to their scale, monitoring and managing forests takes a lot of work. Forestry robotics offers the potential for enabling efficient and sustainable foresting practices through automation. Despite increasing interest in this field, the scarcity of robotics datasets and benchmarks in forest environments is hampering progress in this domain. In this paper, we present a real-world, longitudinal dataset for forestry robotics that enables the development and comparison of approaches for various relevant applications, ranging from semantic interpretation to estimating traits relevant to forestry management. The dataset consists of multiple recordings of the same plots in a forest in Switzerland during three different growth periods. We recorded the data with a mobile 3D LiDAR scanning setup. Additionally, we provide semantic annotations of trees, shrubs, and ground, instance-level annotations of trees, as well as more fine-grained annotations of tree stems and crowns. Furthermore, we provide reference field measurements of traits relevant to forestry management for a subset of the trees. Together with the data, we also provide open-source baseline panoptic segmentation and tree trait estimation approaches to enable the community to bootstrap further research and simplify comparisons in this domain.

Details

ICRA Conference 2025 Conference Paper

Ground-Aware Automotive Radar Odometry

  • Daniel Casado Herraez
  • Franz Kaschner
  • Matthias Zeller
  • Dominik Muhle
  • Jens Behley
  • Michael Heidingsfeld
  • Daniel Cremers
  • Cyrill Stachniss

Odometry is crucial for the navigation of autonomous vehicles in unknown environments. While cameras and LiDARs are commonly used to estimate the ego-motion of a vehicle, these sensors face limitations under bad lighting and severe weather conditions. Automotive radars overcome these challenges, but radar point clouds are generally sparse and noisy, making it difficult to identify useful features within a radar scan. In this paper, we address the problem of ego-motion estimation using a single automotive radar sensor. We propose a simple, yet effective, heuristic-based method to extract the ground plane from single radar scans and perform ground plane matching between consecutive scans. Additionally, we perform a windowed factor-graph optimization of the poses together with the ground plane, improving the accuracy of the pose estimation. We put our work to the test using the 4DRadarDataset. Our findings illustrate the state-of-the-art performance of our odometry approach compared to existing alternatives that use radar point clouds.

Details

ICRA Conference 2025 Conference Paper

Improving Indoor Localization Accuracy by Using an Efficient Implicit Neural Map Representation

  • Haofei Kuang
  • Yue Pan 0009
  • Xingguang Zhong
  • Louis Wiesmann
  • Jens Behley
  • Cyrill Stachniss

Globally localizing a mobile robot in a known map is often a foundation for enabling robots to navigate and operate autonomously. In indoor environments, traditional Monte Carlo localization based on occupancy grid maps is considered the gold standard, but its accuracy is limited by the representation capabilities of the occupancy grid map. In this paper, we address the problem of building an effective map representation that allows to accurately perform probabilistic global localization. To this end, we propose an implicit neural map representation that is able to capture positional and directional geometric features from 2D LiDAR scans to efficiently represent the environment and learn a neural network that is able to predict both, the non-projective signed distance and a direction-aware projective distance for an arbitrary point in the mapped environment. This combination of neural map representation with a lightweight neural network allows us to design an efficient observation model within a conventional Monte Carlo localization framework for pose estimation of a robot in real time. We evaluated our approach to indoor localization on a publicly available dataset for global localization and the experimental results indicate that our approach is able to more accurately localize a mobile robot than other localization approaches employing occupancy or existing neural map representations. In contrast to other approaches employing an implicit neural map representation for 2D LiDAR localization, our approach allows to perform real-time pose tracking after convergence and near real-time global localization. The code of our approach is available at: https://github.com/PRBonn/enm-mcl.

Details

IROS Conference 2025 Conference Paper

Zero-Shot Semantic Segmentation for Robots in Agriculture

  • Yue Linn Chong
  • Lucas Nunes
  • Federico Magistri
  • Xingguang Zhong
  • Jens Behley
  • Cyrill Stachniss

Conventional crop production, which is essential for providing food, feed, fuel, and fiber for our society, relies heavily on harmful herbicides to control weeds. Instead, agricultural robots could remove weeds more sustainably. However, these robots require a generalizable perception system that can locate weeds, enabling automatic removal of weeds. Specifically, they need to perform crop-weed semantic segmentation, which locates and distinguishes between the crop and the weed plants with pixel-level resolution. However, most existing crop-weed semantic segmentation methods are fully supervised and require expensive and labor-intensive pixel-wise labeling of the training data. To avoid the costly labeling process, we address the problem of unsupervised crop-weed segmentation in this paper. Unlike previous approaches, we leverage the idea that weeds are "weird" plants that occur less frequently and are highly variable in appearance, and reframe the problem as an anomaly segmentation problem. We propose an approach to segment weeds as anomalous plants by categorizing plants in the feature space of a pretrained foundation model. Our approach curates a bag-of-features representation of crop features and models the manifold of crop plants as hyperspheres. During inference, it classifies vegetation segments of the image with features within this manifold as crop plants and all other plants as weeds. Our experiments show that our zero-shot anomaly segmentation method can perform crop-weed segmentation on several datasets from real crop fields.

Details

IROS Conference 2024 Conference Paper

BonnBeetClouds3D: A Dataset Towards Point Cloud-Based Organ-Level Phenotyping of Sugar Beet Plants Under Real Field Conditions

  • Elias Marks
  • Jonas Bömer
  • Federico Magistri
  • Anurag Sag
  • Jens Behley
  • Cyrill Stachniss

Agricultural production is facing challenges in the next decades induced by climate change and the need for more sustainability by reducing its impact on the environment. Advances in field management through robotic intervention, monitoring of crops by autonomous unmanned aerial vehicles (UAVs) supporting breeding of novel and more resilient crop varieties can help to address these challenges. The analysis of plant traits is called phenotyping and is an essential activity in plant breeding; it however involves a great amount of manual labor. With this paper, we provide means to better tackle the problems of instance segmentation to support robotic intervention and automatic fine-grained, organ-level geometric analysis needed for precision phenotyping. As the availability of real-world data in this domain is relatively scarce, we provide a novel dataset that was acquired using UAVs capturing high-resolution images of real breeding trials containing 48 plant varieties and therefore covering a relevant morphological and appearance spectrum. This enables the development of approaches for instance segmentation and autonomous phenotyping that generalize well to different plant varieties. Based on overlapping high-resolution images taken from multiple viewing angles, we provide photogrammetric dense point clouds and provide detailed and accurate point-wise labels for plants, leaves, and salient points as the tip and the base in 3D. Additionally, we include measurements of phenotypic traits performed by experts from the German Federal Plant Variety Office on the real plants, allowing the evaluation of new approaches not only on segmentation and keypoint detection but also directly on actual traits. The provided labeled point clouds enable finegrained plant analysis and support further progress in the development of automatic phenotyping approaches, but also enable further research in surface reconstruction, point cloud completion, and semantic interpretation of point clouds.

Details

ICRA Conference 2024 Conference Paper

Efficient and Accurate Transformer-Based 3D Shape Completion and Reconstruction of Fruits for Agricultural Robots

  • Federico Magistri
  • Rodrigo Marcuzzi
  • Elias Marks
  • Matteo Sodano
  • Jens Behley
  • Cyrill Stachniss

Robots that operate in agricultural environments need a robust perception system that can deal with occlusions, which are naturally present in agricultural scenarios. In this paper, we address the problem of estimating 3D shapes of fruits when only partial observations are available. Generally speaking, such a shape completion can be realized by exploiting prior knowledge about the geometry of the fruit. This is typically done by template matching using traditional optimization algorithms, which are slow but accurate, or by encoding such knowledge into the weights of a neural network, leading to faster but often less accurate estimates. Our approach combines the best of both worlds. It exploits the benefit of having a template representing our object of interest with the advantages of using a neural network to learn how to deform a template. Our experimental evaluation demonstrates that our approach yields accurate estimation at a competitively low inference time in challenging greenhouse environments.

Details

IROS Conference 2024 Conference Paper

HeLiMOS: A Dataset for Moving Object Segmentation in 3D Point Clouds From Heterogeneous LiDAR Sensors

  • Hyungtae Lim
  • Seoyeon Jang
  • Benedikt Mersch
  • Jens Behley
  • Hyun Myung
  • Cyrill Stachniss

Moving object segmentation (MOS) using a 3D light detection and ranging (LiDAR) sensor is crucial for scene understanding and identification of moving objects. Despite the availability of various types of 3D LiDAR sensors in the market, MOS research still predominantly focuses on 3D point clouds from mechanically spinning omnidirectional LiDAR sensors. Thus, we are, for example, lacking a dataset with MOS labels for point clouds from solid-state LiDAR sensors which have irregular scanning patterns. In this paper, we present a labeled dataset, called HeLiMOS, that enables to test MOS approaches on four heterogeneous LiDAR sensors, including two solid-state LiDAR sensors. Furthermore, we introduce a novel automatic labeling method to substantially reduce the labeling effort required from human annotators. To this end, our framework exploits an instance-aware static map building approach and tracking-based false label filtering. Finally, we provide experimental results regarding the performance of commonly used state-of-the-art MOS approaches on HeLiMOS that suggest a new direction for a sensor-agnostic MOS, which generally works regardless of the type of LiDAR sensors used to capture 3D point clouds. Our dataset is available at https://sites.google.com/view/helimos.

Details

ICRA Conference 2024 Conference Paper

Radar Tracker: Moving Instance Tracking in Sparse and Noisy Radar Point Clouds

  • Matthias Zeller
  • Daniel Casado Herraez
  • Jens Behley
  • Michael Heidingsfeld
  • Cyrill Stachniss

Robots and autonomous vehicles should be aware of what happens in their surroundings. The segmentation and tracking of moving objects are essential for reliable path planning, including collision avoidance. We investigate this estimation task for vehicles using radar sensing. We address moving instance tracking in sparse radar point clouds to enhance scene interpretation. We propose a learning-based radar tracker incorporating temporal offset predictions to enable direct center-based association and enhance segmentation performance by including additional motion cues. We implement attention-based tracking for sparse radar scans to include appearance features and enhance performance. The final association combines geometric and appearance features to overcome the limitations of center-based tracking to associate instances reliably. Our approach shows an improved performance on the moving instance tracking benchmark of the RadarScenes dataset compared to the current state of the art.

Details

ICRA Conference 2024 Conference Paper

Tree Instance Segmentation and Traits Estimation for Forestry Environments Exploiting LiDAR Data Collected by Mobile Robots

  • Meher V. R. Malladi
  • Tiziano Guadagnino
  • Luca Lobefaro
  • Matías Mattamala
  • Holger Griess
  • Janine Schweier
  • Nived Chebrolu
  • Maurice F. Fallon

Forests play a crucial role in our ecosystems, functioning as carbon sinks, climate stabilizers, biodiversity hubs, and sources of wood. By the very nature of their scale, monitoring and maintaining forests is a challenging task. Robotics in forestry can have the potential for substantial automation toward efficient and sustainable foresting practices. In this paper, we address the problem of automatically producing a forest inventory by exploiting LiDAR data collected by a mobile platform. To construct an inventory, we first extract tree instances from point clouds. Then, we process each instance to extract forestry inventory information. Our approach provides the per-tree geometric trait of "diameter at breast height" together with the individual tree locations in a plot. We validate our results against manual measurements collected by foresters during field trials. Our experiments show strong segmentation and tree trait estimation performance, underlining the potential for automating forestry services. Results furthermore show a superior performance compared to the popular baseline methods used in this domain.

Details

IROS Conference 2023 Conference Paper

Constructing Metric-Semantic Maps Using Floor Plan Priors for Long-Term Indoor Localization

  • Nicky Zimmerman
  • Matteo Sodano
  • Elias Marks
  • Jens Behley
  • Cyrill Stachniss

Object-based maps are relevant for scene under-standing since they integrate geometric and semantic information of the environment, allowing autonomous robots to robustly localize and interact with on objects. In this paper, we address the task of constructing a metric-semantic map for the purpose of long-term object-based localization. We exploit 3D object detections from monocular RGB frames for both, the object-based map construction, and for globally localizing in the constructed map. To tailor the approach to a target environment, we propose an efficient way of generating 3D annotations to finetune the 3D object detection model. We evaluate our map construction in an office building, and test our long-term localization approach on challenging sequences recorded in the same environment over nine months. The experiments suggest that our approach is suitable for constructing metric-semantic maps, and that our localization approach is robust to long-term changes. Both, the mapping algorithm and the localization pipeline can run online on an onboard computer. We release an open-source C++/ros implementation of our approach.

Details

ICRA Conference 2023 Conference Paper

Fruit Tracking Over Time Using High-Precision Point Clouds

  • Alessandro Riccardi
  • Shane Kelly
  • Elias Marks
  • Federico Magistri
  • Tiziano Guadagnino
  • Jens Behley
  • Maren Bennewitz
  • Cyrill Stachniss

Monitoring the traits of plants and fruits is a fundamental task in horticulture. With accurate measurements, farmers can predict the yield of their crops and use this information for making informed management decisions, and breeders can use it for variety selection. Agricultural robotic applications promise to automate this monitoring task. In this paper, we address the problem of monitoring fruit growth and investigate the matching of fruits recorded in commercial greenhouses at different growth stages based on data recorded from terrestrial laser scanners. This is challenging as fruits appear highly similar, change over time, and are subject to severe occlusions. We first propose a fruit descriptor, which captures the topology of the fruit surroundings to facilitate the matching between different points in time. We capture and describe the relationship between a fruit and its neighbors such that our descriptors are less affected by the growth over time. Furthermore, we define a matching cost function and use an optimal assignment algorithm to match the fruit observations taken in different weeks. The experiments show that our descriptor achieves a high spatio-temporal matching accuracy, which is superior to the commonly used geometric point cloud descriptors.

Details

ICRA Conference 2023 Conference Paper

Hierarchical Approach for Joint Semantic, Plant Instance, and Leaf Instance Segmentation in the Agricultural Domain

  • Gianmarco Roggiolani
  • Matteo Sodano
  • Tiziano Guadagnino
  • Federico Magistri
  • Jens Behley
  • Cyrill Stachniss

Plant phenotyping is a central task in agriculture, as it describes plants' growth stage, development, and other relevant quantities. Robots can help automate this process by accurately estimating plant traits such as the number of leaves, leaf area, and the plant size. In this paper, we address the problem of joint semantic, plant instance, and leaf instance segmentation of crop fields from RGB data. We propose a single convolutional neural network that addresses the three tasks simultaneously, exploiting their underlying hierarchical structure. We introduce task-specific skip connections, which our experimental evaluation proves to be more beneficial than the usual schemes. We also propose a novel automatic post-processing, which explicitly addresses the problem of spatially close instances, common in the agricultural domain because of overlapping leaves. Our architecture simultaneously tackles these problems jointly in the agricultural context. Previous works either focus on plant or leaf segmentation, or do not optimise for semantic segmentation. Results show that our system has superior performance compared to state-of-the-art approaches, while having a reduced number of parameters and is operating at camera frame rate.

Details

ICRA Conference 2023 Conference Paper

On Domain-Specific Pre- Training for Effective Semantic Perception in Agricultural Robotics

  • Gianmarco Roggiolani
  • Federico Magistri
  • Tiziano Guadagnino
  • Jan Weyler
  • Giorgio Grisetti
  • Cyrill Stachniss
  • Jens Behley

Agricultural robots have the prospect to enable more efficient and sustainable agricultural production of food, feed, and fiber. Perception of crops and weeds is a central component of agricultural robots that aim to monitor fields and assess the plants as well as their growth stage in an automatic manner. Semantic perception mostly relies on deep learning using supervised approaches, which require time and qualified workers to label fairly large amounts of data. In this paper, we look into the problem of reducing the amount of labels without compromising the final segmentation performance. For robots operating in the field, pre-training networks in a supervised way is already a popular method to reduce the number of required labeled images. We investigate the possibility of pre-training in a self-supervised fashion using data from the target domain. To better exploit this data, we propose a set of domain-specific augmentation strategies. We evaluate our pre-training on semantic segmentation and leaf instance segmentation, two important tasks in our domain. The experimental results suggest that pre-training with domain-specific data paired with our data augmentation strategy leads to superior performance compared to commonly used pre-trainings. Furthermore, the pre-trained networks obtain similar performance to the fully supervised with less labeled data.

Details

IROS Conference 2023 Conference Paper

Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots

  • Yue Pan 0009
  • Federico Magistri
  • Thomas Läbe
  • Elias Marks
  • Claus Smitt
  • Chris McCool
  • Jens Behley
  • Cyrill Stachniss

Monitoring plants and fruits at high resolution play a key role in the future of agriculture. Accurate 3D information can pave the way to a diverse number of robotic applications in agriculture ranging from autonomous harvesting to precise yield estimation. Obtaining such 3D information is non-trivial as agricultural environments are often repetitive and cluttered, and one has to account for the partial observability of fruit and plants. In this paper, we address the problem of jointly estimating complete 3D shapes of fruit and their pose in a 3D multi-resolution map built by a mobile robot. To this end, we propose an online multi-resolution panoptic mapping system where regions of interest are represented with a higher resolution. We exploit data to learn a general fruit shape representation that we use at inference time together with an occlusion-aware differentiable rendering pipeline to complete partial fruit observations and estimate the 7 DoF pose of each fruit in the map. The experiments presented in this paper, evaluated both in the controlled environment and in a commercial greenhouse, show that our novel algorithm yields higher completion and pose estimation accuracy than existing methods, with an improvement of 41 % in completion accuracy and 52 % in pose estimation accuracy while keeping a low inference time of 0. 6 s in average.

Details

ICRA Conference 2023 Conference Paper

Radar Velocity Transformer: Single-scan Moving Object Segmentation in Noisy Radar Point Clouds

  • Matthias Zeller
  • Vardeep S. Sandhu
  • Benedikt Mersch
  • Jens Behley
  • Michael Heidingsfeld
  • Cyrill Stachniss

The awareness about moving objects in the surroundings of a self-driving vehicle is essential for safe and reliable autonomous navigation. The interpretation of LiDAR and camera data achieves exceptional results but typically requires to accumulate and process temporal sequences of data in order to extract motion information. In contrast, radar sensors, which are already installed in most recent vehicles, can overcome this limitation as they directly provide the Doppler velocity of the detections and, hence incorporate instantaneous motion information within a single measurement. In this paper, we tackle the problem of moving object segmentation in noisy radar point clouds. We also consider differentiating parked from moving cars, to enhance scene understanding. Instead of exploiting temporal dependencies to identify moving objects, we develop a novel transformer-based approach to perform single-scan moving object segmentation in sparse radar scans accurately. The key to our Radar Velocity Transformer is to incorporate the valuable velocity information throughout each module of the network, thereby enabling the precise segmentation of moving and non-moving objects. Additionally, we propose a transformer-based upsampling, which enhances the performance by adaptively combining information and over-coming the limitation of interpolation of sparse point clouds. Finally, we create a new radar moving object segmentation benchmark based on the RadarScenes dataset and compare our approach to other state-of-the-art methods. Our network runs faster than the frame rate of the sensor and shows superior segmentation results using only single-scan radar data.

Details

ICRA Conference 2023 Conference Paper

Robust Double-Encoder Network for RGB-D Panoptic Segmentation

  • Matteo Sodano
  • Federico Magistri
  • Tiziano Guadagnino
  • Jens Behley
  • Cyrill Stachniss

Perception is crucial for robots that act in real-world environments, as autonomous systems need to see and understand the world around them to act properly. Panoptic segmentation provides an interpretation of the scene by computing a pixelwise semantic label together with instance IDs. In this paper, we address panoptic segmentation using RGB-D data of indoor scenes. We propose a novel encoder-decoder neural network that processes RGB and depth separately through two encoders. The features of the individual encoders are progressively merged at different resolutions, such that the RGB features are enhanced using complementary depth information. We propose a novel merging approach called ResidualExcite, which reweighs each entry of the feature map according to its importance. With our double-encoder architecture, we are robust to missing cues. In particular, the same model can train and infer on RGB-D, RGB-only, and depth-only input data, without the need to train specialized models. We evaluate our method on publicly available datasets and show that our approach achieves superior results compared to other common approaches for panoptic segmentation.

Details

ICRA Conference 2023 Conference Paper

SHINE-Mapping: Large-Scale 3D Mapping Using Sparse Hierarchical Implicit Neural Representations

  • Xingguang Zhong
  • Yue Pan 0009
  • Jens Behley
  • Cyrill Stachniss

Accurate mapping of large-scale environments is an essential building block of most outdoor autonomous systems. Challenges of traditional mapping methods include the balance between memory consumption and mapping accuracy. This paper addresses the problem of achieving large-scale 3D reconstruction using implicit representations built from 3D LiDAR measurements. We learn and store implicit features through an octree-based, hierarchical structure, which is sparse and extensible. The implicit features can be turned into signed distance values through a shallow neural network. We leverage binary cross entropy loss to optimize the local features with the 3D measurements as supervision. Based on our implicit representation, we design an incremental mapping system with regularization to tackle the issue of forgetting in continual learning. Our experiments show that our 3D reconstructions are more accurate, complete, and memory-efficient than current state-of-the-art 3D mapping methods.

Details

ICRA Conference 2022 Conference Paper

Retriever: Point Cloud Retrieval in Compressed 3D Maps

  • Louis Wiesmann
  • Rodrigo Marcuzzi
  • Cyrill Stachniss
  • Jens Behley

Most autonomous driving and robotic applications require retrieving map data around the vehicle's current location. Those maps can cover large areas and are often stored in a compressed form to save memory and allow for efficient transmission. In this paper, we address the problem of place recognition in a compressed point cloud map. To this end, we propose a novel deep neural network architecture that directly operates on a compressed feature representation produced by a compression encoder. This enables us to bypass compute-heavy decompression of the map and exploits the compact as well as descriptive nature of the compressed features. Additionally, we propose an alternative to the commonly used NetVLAD layer to aggregate local descriptors. Here, we utilize an attention mechanism between local features and a latent code. Our experiments suggest that this produces a more descriptive feature representation of the point clouds for place recognition. We experimentally validate all architectural choices we made by our ablation studies and compare our performance to other state-of-the-art baselines on two commonly used datasets.

Details

IROS Conference 2022 Conference Paper

Robust Onboard Localization in Changing Environments Exploiting Text Spotting

  • Nicky Zimmerman
  • Louis Wiesmann
  • Tiziano Guadagnino
  • Thomas Läbe
  • Jens Behley
  • Cyrill Stachniss

Robust localization in a given map is a crucial component of most autonomous robots. In this paper, we address the problem of localizing in an indoor environment that changes and where prominent structures have no correspondence in the map built at a different point in time. To overcome the discrepancy between the map and the observed environment caused by such changes, we exploit human-readable localization cues to assist localization. These cues are readily available in most facilities and can be detected using RGB camera images by utilizing text spotting. We integrate these cues into a Monte Carlo localization framework using a particle filter that operates on 2D LiDAR scans and camera data. By this, we provide a robust localization solution for environments with structural changes and dynamics by humans walking. We evaluate our localization framework on multiple challenging indoor scenarios in an office environment. The experiments suggest that our approach is robust to structural changes and can run on an onboard computer. We release an open source implementation of our approach 1 1 https://github.com/PRBonn/tmcl, which uses off-the-shelf text spotting, written in C++ with a ROS wrapper.

Details

ICRA Conference 2021 Conference Paper

A Benchmark for LiDAR-based Panoptic Segmentation based on KITTI

  • Jens Behley
  • Andres Milioto
  • Cyrill Stachniss

Panoptic segmentation is the recently introduced task that tackles semantic segmentation and instance segmentation jointly [18]. In this paper, we present an extension of SemanticKITTI [1], a large-scale dataset providing dense point-wise semantic labels for all sequences of the KITTI Odometry Benchmark [10]. This extension enables training and evaluation of LiDAR-based panoptic segmentation. We provide the data and discuss the processing steps needed to enrich a given semantic annotation with temporally consistent instance information, i. e. , instance information that supplements the semantic labels and identifies the same instance over sequences of LiDAR point clouds. Additionally, we present two strong baselines that combine state-of-the-art LiDAR-based semantic segmentation approaches with a state-of-the-art detector enriching the segmentation with instance information and that allow other researchers to compare their approaches against. We believe that our extension of SemanticKITTI with strong baselines enables the creation of novel algorithms for LiDAR-based panoptic segmentation as much as it has for the original semantic segmentation and semantic scene completion tasks. Data, code, and an online evaluation service using a hidden test set are publicly available at http://semantic-kitti.org.

Details

IROS Conference 2021 Conference Paper

Improving Monocular Depth Estimation by Semantic Pre-training

  • Peter Rottmann
  • Thorbjörn Posewsky
  • Andres Milioto
  • Cyrill Stachniss
  • Jens Behley

Knowing the distance to nearby objects is crucial for autonomous cars to navigate safely in everyday traffic. In this paper, we investigate monocular depth estimation, which advanced substantially within the last years and is providing increasingly more accurate results while only requiring a single camera image as input. In line with recent work, we use an encoder-decoder structure with so-called packing layers to estimate depth values in a self-supervised fashion. We propose integrating a joint pre-training of semantic segmentation plus depth estimation on a dataset providing semantic labels. By using a separate semantic decoder that is only needed for pre-training, we can keep the network comparatively small. Our extensive experimental evaluation shows that the addition of such pre-training improves the depth estimation performance substantially. Finally, we show that we achieve competitive performance on the KITTI dataset despite using a much smaller and more efficient network.

Details

ICRA Conference 2021 Conference Paper

Poisson Surface Reconstruction for LiDAR Odometry and Mapping

  • Ignacio Vizzo
  • Xieyuanli Chen
  • Nived Chebrolu
  • Jens Behley
  • Cyrill Stachniss

Accurately localizing in and mapping an environment are essential building blocks of most autonomous systems. In this paper, we present a novel approach for LiDAR odometry and mapping, focusing on improving the mapping quality and at the same time estimating the pose of the vehicle. Our approach performs frame-to-mesh ICP, but in contrast to other SLAM approaches, we represent the map as a triangle mesh computed via Poisson surface reconstruction. We perform the surface reconstruction in a sliding window fashion over a sequence of past scans. In this way, we obtain accurate local maps that are well suited for registration and can also be combined into a global map. This enables us to build a 3D map showing more geometric details than common mapping approaches relying on a truncated signed distance function or surfels. Our experimental evaluation shows quantitatively and qualitatively that our maps offer higher geometric accuracies than these other map representations. We also show that our maps are compact and can be used for LiDAR-based odometry estimation with a novel ray-casting-based data association.

Details

ICRA Conference 2021 Conference Paper

Range Image-based LiDAR Localization for Autonomous Vehicles

  • Xieyuanli Chen
  • Ignacio Vizzo
  • Thomas Läbe
  • Jens Behley
  • Cyrill Stachniss

Robust and accurate, map-based localization is crucial for autonomous mobile systems. In this paper, we exploit range images generated from 3D LiDAR scans to address the problem of localizing mobile robots or autonomous cars in a map of a large-scale outdoor environment represented by a triangular mesh. We use the Poisson surface reconstruction to generate the mesh-based map representation. Based on the range images generated from the current LiDAR scan and the synthetic rendered views from the mesh-based map, we propose a new observation model and integrate it into a Monte Carlo localization framework, which achieves better localization performance and generalizes well to different environments. We test the proposed localization approach on multiple datasets collected in different environments with different LiDAR scanners. The experimental results show that our method can reliably and accurately localize a mobile system in different environments and operate online at the LiDAR sensor frame rate to track the vehicle pose.

Details

ICRA Conference 2021 Conference Paper

Towards In-Field Phenotyping Exploiting Differentiable Rendering with Self-Consistency Loss

  • Federico Magistri
  • Nived Chebrolu
  • Jens Behley
  • Cyrill Stachniss

In modern agriculture, measuring phenotypic traits helps breeders monitor plant growth, increase yield, and provide food, feed, and fiber. Traditional phenotyping requires intensive manual work, partially being intrusive. In this paper, we investigate the challenge of measuring phenotypic traits in an automated fashion through mobile robots operating in field environments. In particular, we want to measure plants from images acquired by mobile robots instead of using data from a static scanning environment. We propose to use a differentiable rendering approach to deform a generic 3D template of a plant to fit the observation recorded by a robot while ensuring a coherent deformation of the plant template. The experiments presented in this paper suggest that our approach allows for 3D reconstruction of different plant species at different growth stages using single images. From that model, we can compute important phenotypic traits, such as the leaf area index.

Details

IROS Conference 2020 Conference Paper

Domain Transfer for Semantic Segmentation of LiDAR Data using Deep Neural Networks

  • Ferdinand Langer
  • Andres Milioto
  • Alexandre Haag
  • Jens Behley
  • Cyrill Stachniss

Inferring semantic information towards an understanding of the surrounding environment is crucial for autonomous vehicles to drive safely. Deep learning-based segmentation methods can infer semantic information directly from laser range data, even in the absence of other sensor modalities such as cameras. In this paper, we address improving the generalization capabilities of such deep learning models to range data that was captured using a different sensor and in situations where no labeled data is available for the new sensor setup. Our approach assists the domain transfer of a LiDAR-only semantic segmentation model to a different sensor and environment exploiting existing geometric mapping systems. To this end, we fuse sequential scans in the source dataset into a dense mesh and render semi-synthetic scans that match those of the target sensor setup. Unlike simulation, this approach provides a real-to-real transfer of geometric information and delivers additionally more accurate remission information. We implemented and thoroughly tested our approach by transferring semantic scans between two different real-world datasets with different sensor setups. Our experiments show that we can improve the segmentation performance substantially with zero manual re-labeling. This approach solves the number one feature request since we released our semantic segmentation library LiDAR-bonnetal [18].

Details

IROS Conference 2020 Conference Paper

Learning an Overlap-based Observation Model for 3D LiDAR Localization

  • Xieyuanli Chen
  • Thomas Läbe
  • Lorenzo Nardi
  • Jens Behley
  • Cyrill Stachniss

Localization is a crucial capability for mobile robots and autonomous cars. In this paper, we address learning an observation model for Monte-Carlo localization using 3D LiDAR data. We propose a novel, neural network-based observation model that computes the expected overlap of two 3D LiDAR scans. The model predicts the overlap and yaw angle offset between the current sensor reading and virtual frames generated from a pre-built map. We integrate this observation model into a Monte-Carlo localization framework and tested it on urban datasets collected with a car in different seasons. The experiments presented in this paper illustrate that our method can reliably localize a vehicle in typical urban environments. We furthermore provide comparisons to a beam-endpoint and a histogram-based method indicating a superior global localization performance of our method with fewer particles.

Details

IROS Conference 2020 Conference Paper

LiDAR Panoptic Segmentation for Autonomous Driving

  • Andres Milioto
  • Jens Behley
  • Chris McCool
  • Cyrill Stachniss

Truly autonomous driving without the need for human intervention can only be attained when self-driving cars fully understand their surroundings. Most of these vehicles rely on a suite of active and passive sensors. LiDAR sensors are a cornerstone in most of these hardware stacks, and leveraging them as a complement to other passive sensors such as RGB cameras is an enticing goal. Understanding the semantic class of each point in a LiDAR sweep is important, as well as knowing to which instance of that class it belongs to. To this end, we present a novel, single-stage, and real-time capable panoptic segmentation approach using a shared encoder with a semantic and instance decoder. We leverage the geometric information of the LiDAR scan to perform a novel, distance- aware tri-linear upsampling, which allows our approach to use larger output strides than using transpose convolutions leading to substantial savings in computation time. Our experimental evaluation and ablation studies for each module show that combining our geometric and semantic embeddings with our learned, variable instance thresholds, a category-specific loss, and the novel trilinear upsampling module leads to higher panoptic quality. We will release the code of our approach in our LiDAR processing library LiDAR-Bonnetal [27].

Details

IROS Conference 2019 Conference Paper

RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation

  • Andres Milioto
  • Ignacio Vizzo
  • Jens Behley
  • Cyrill Stachniss

Perception in autonomous vehicles is often carried out through a suite of different sensing modalities. Given the massive amount of openly available labeled RGB data and the advent of high-quality deep learning algorithms for image-based recognition, high-level semantic perception tasks are pre-dominantly solved using high-resolution cameras. As a result of that, other sensor modalities potentially useful for this task are often ignored. In this paper, we push the state of the art in LiDAR-only semantic segmentation forward in order to provide another independent source of semantic information to the vehicle. Our approach can accurately perform full semantic segmentation of LiDAR point clouds at sensor frame rate. We exploit range images as an intermediate representation in combination with a Convolutional Neural Network (CNN) exploiting the rotating LiDAR sensor model. To obtain accurate results, we propose a novel post-processing algorithm that deals with problems arising from this intermediate representation such as discretization errors and blurry CNN outputs. We implemented and thoroughly evaluated our approach including several comparisons to the state of the art. Our experiments show that our approach outperforms state-of-the-art approaches, while still running online on a single embedded GPU. The code can be accessed at https://github.com/PRBonn/lidar-bonnetal.

Details

IROS Conference 2019 Conference Paper

ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals

  • Emanuele Palazzolo
  • Jens Behley
  • Philipp Lottes
  • Philippe Giguère
  • Cyrill Stachniss

Mapping and localization are essential capabilities of robotic systems. Although the majority of mapping systems focus on static environments, the deployment in real-world situations requires them to handle dynamic objects. In this paper, we propose an approach for an RGB-D sensor that is able to consistently map scenes containing multiple dynamic elements. For localization and mapping, we employ an efficient direct tracking on the truncated signed distance function (TSDF) and leverage color information encoded in the TSDF to estimate the pose of the sensor. The TSDF is efficiently represented using voxel hashing, with most computations parallelized on a GPU. For detecting dynamics, we exploit the residuals obtained after an initial registration, together with the explicit modeling of free space in the model. We evaluate our approach on existing datasets, and provide a new dataset showing highly dynamic scenes. These experiments show that our approach often surpass other state-of-the-art dense SLAM methods. We make available our dataset with the ground truth for both the trajectory of the RGB-D sensor obtained by a motion capture system and the model of the static environment using a high-precision terrestrial laser scanner. Finally, we release our approach as open source code.

Details

IROS Conference 2019 Conference Paper

SuMa++: Efficient LiDAR-based Semantic SLAM

  • Xieyuanli Chen
  • Andres Milioto
  • Emanuele Palazzolo
  • Philippe Giguère
  • Jens Behley
  • Cyrill Stachniss

Reliable and accurate localization and mapping are key components of most autonomous systems. Besides geometric information about the mapped environment, the semantics plays an important role to enable intelligent navigation behaviors. In most realistic environments, this task is particularly complicated due to dynamics caused by moving objects, which can corrupt the mapping step or derail localization. In this paper, we propose an extension of a recently published surfel-based mapping approach exploiting three-dimensional laser range scans by integrating semantic information to facilitate the mapping process. The semantic information is efficiently extracted by a fully convolutional neural network and rendered on a spherical projection of the laser range data. This computed semantic segmentation results in point-wise labels for the whole scan, allowing us to build a semantically-enriched map with labeled surfels. This semantic map enables us to reliably filter moving objects, but also improve the projective scan matching via semantic constraints. Our experimental evaluation on challenging highways sequences from KITTI dataset with very few static structures and a large amount of moving cars shows the advantage of our semantic SLAM approach in comparison to a purely geometric, state-of-the-art approach.

Details

IROS Conference 2018 Conference Paper

Joint Stem Detection and Crop-Weed Classification for Plant-Specific Treatment in Precision Farming

  • Philipp Lottes
  • Jens Behley
  • Nived Chebrolu
  • Andres Milioto
  • Cyrill Stachniss

Applying agrochemicals is the default procedure for conventional weed control in crop production, but has negative impacts on the environment. Robots have the potential to treat every plant in the field individually and thus can reduce the required use of such chemicals. To achieve that, robots need the ability to identify crops and weeds in the field and must additionally select effective treatments. While certain types of weed can be treated mechanically, other types need to be treated by (selective) spraying. In this paper, we present an approach that provides the necessary information for effective plant-specific treatment. It outputs the stem location for weeds, which allows for mechanical treatments, and the covered area of the weed for selective spraying. Our approach uses an end-to-end trainable fully convolutional network that simultaneously estimates stem positions as well as the covered area of crops and weeds. It jointly learns the class-wise stem detection and the pixel-wise semantic segmentation. Experimental evaluations on different real-world datasets show that our approach is able to reliably solve this problem. Compared to state-of-the-art approaches, our approach not only substantially improves the stem detection accuracy, i. e. , distinguishing crop and weed stems, but also provides an improvement in the semantic segmentation performance.

Details

ICRA Conference 2015 Conference Paper

Efficient radius neighbor search in three-dimensional point clouds

  • Jens Behley
  • Volker Steinhage
  • Armin B. Cremers

Finding all neighbors of a point inside a given radius is an integral part in many approaches using three-dimensional laser range data. We present novel insights to significantly improve the runtime performance of radius neighbor search using octrees. Our contributions are as follows: (1) We propose an index-based organization of the point cloud such that we can efficiently store start and end indexes of points inside every octant and (2) exploiting this representation, we can use pruning of irrelevant subtrees in the traversal to facilitate highly efficient radius neighbor search. We show significant runtime improvements of our proposed octree representation over state-of-the-art neighbor search implementations on three different urban datasets.

Details

IROS Conference 2013 Conference Paper

Laser-based segment classification using a mixture of bag-of-words

  • Jens Behley
  • Volker Steinhage
  • Armin B. Cremers

In this paper, we propose a segment-based object detection approach using laser range data. Our detection approach is built up of three stages: First, a hierarchical segmentation approach generates a hierarchy of coarse-to-fine segments to reduce the impact of over- and under-segmentation in later stages. Next, we employ a learned mixture model to classify all segments. The model combines multiple softmax regression classifiers learned on specific bag-of-word representations using different parameterizations of a descriptor. In the final stage, we filter irrelevant and duplicate detections using a greedy method in consideration of the segment hierarchy. We experimentally evaluate our approach on recently published real-world datasets to detect pedestrians, cars, and cyclists.

Details

ICRA Conference 2012 Conference Paper

Performance of histogram descriptors for the classification of 3D laser range data in urban environments

  • Jens Behley
  • Volker Steinhage
  • Armin B. Cremers

The selection of suitable features and their parameters for the classification of three-dimensional laser range data is a crucial issue for high-quality results. In this paper we compare the performance of different histogram descriptors and their parameters on three urban datasets recorded with various sensors-sweeping SICK lasers, tilting SICK lasers and a Velodyne 3D laser range scanner. These descriptors are 1D, 2D, and 3D histograms capturing the distribution of normals or points around a query point. We also propose a novel histogram descriptor, which relies on the spectral values in different scales. We argue that choosing a larger support radius and a z-axis based global reference frame/axis can boost the performance of all kinds of investigated classification models significantly. The 3D histograms relying on the point distribution, normal orientations, or spectral values, turned out to be the best choice for the classification in urban environments.

Details

ICRA Conference 2011 Conference Paper

Person tracking in three-dimensional laser range data with explicit occlusion adaption

  • Florian Schöler
  • Jens Behley
  • Volker Steinhage
  • Dirk Schulz 0001
  • Armin B. Cremers

This paper presents an approach to exploit the richer information of sensor data provided by 3d laser rangefinders for the purpose of person tracking. Introduced is a method to adapt the observation model of a particle filter, to identify partial and full occlusions of a person, to determine the amount of occlusion behind an obstacle, and the occluding obstacle itself. This is done by tracing rays from positions near the person to the sensor and determining whether the ray hits an obstacle. The laser range data is represented using a voxel grid, which facilitates efficient retrieval and data reduction. As our experiments show, our proposed tracking approach is able to reliably keep track of a person in real-time, even when only partially visible, when moving in uneven terrain, or when the person passes closely another person of different size.

Details

IROS Conference 2010 Conference Paper

Learning to hash logistic regression for fast 3D scan point classification

  • Jens Behley
  • Kristian Kersting
  • Dirk Schulz 0001
  • Volker Steinhage
  • Armin B. Cremers

Segmenting range data into semantic categories has become a more and more active field of research in robotics. In this paper, we advocate to view this task as a problem of fast, large-scale retrieval. Intuitively, given a dataset of millions of labeled scan points and their neighborhoods, we simply search for similar points in the datasets and use the labels of the retrieved ones to predict the labels of a novel point using some local prediction model such as majority vote or logistic regression. However, actually carrying this out requires highly efficient ways of (1) storing millions of scan points in memory and (2) quickly finding similar scan points to a target scan point. In this paper, we propose to address both issues by employing Weiss et al. 's recent spectral hashing. It represents each item in a database by a compact binary code that is constructed so that similar items will have similar binary code words. In turn, similar neighbors have codes within a small Hamming distance of the code for the query. Then, we learn a logistic regression model locally over all points with the same binary code word. Our experiments on real world 3D scans show that the resulting approach, called spectrally hashed logistic regression, can be ultra fast at prediction time and outperforms state-of-the art approaches such as logistic regression and nearest neighbor.

Details
v2026.09.13