Author name cluster

Klaus Dietmayer

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

19 papers

2 author rows

ICRA Conference 2024 Conference Paper

Multimodal Object Query Initialization for 3D Object Detection

Mathijs R. van Geerenstein
Felicia Ruppel
Klaus Dietmayer
Dariu M. Gavrila

3D object detection models that exploit both LiDAR and camera sensor features are top performers in large-scale autonomous driving benchmarks. A transformer is a popular network architecture used for this task, in which so-called object queries act as candidate objects. Initializing these object queries based on current sensor inputs is a common practice. For this, existing methods strongly rely on LiDAR data however, and do not fully exploit image features. Besides, they introduce significant latency. To overcome these limitations we propose EfficientQ3M, an efficient, modular, and multimodal solution for object query initialization for transformer-based 3D object detection models. The proposed initialization method is combined with a "modality-balanced" transformer decoder where the queries can access all sensor modalities throughout the decoder. In experiments, we outperform the state of the art in transformer-based LiDAR object detection on the competitive nuScenes benchmark and showcase the benefits of input-dependent multimodal query initialization, while being more efficient than the available alternatives for LiDAR-camera initialization. The proposed method can be applied with any combination of sensor modalities as input, demonstrating its modularity.

ICRA Conference 2023 Conference Paper

Exploring Navigation Maps for Learning-Based Motion Prediction

Julian Schmidt
Julian Jordan
Franz Gritschneder
Thomas Monninger
Klaus Dietmayer

The prediction of surrounding agents' motion is a key for safe autonomous driving. In this paper, we explore navigation maps as an alternative to the predominant High Definition (HD) maps for learning-based motion prediction. Navigation maps provide topological and geometrical information on road-level, HD maps additionally have centimeter-accurate lane-level information. As a result, HD maps are costly and time-consuming to obtain, while navigation maps with near-global coverage are freely available. We describe an approach to integrate navigation maps into learning-based motion prediction models. To exploit locally available HD maps during training, we additionally propose a model-agnostic method for knowledge distillation. In experiments on the publicly available Argoverse dataset with navigation maps obtained from OpenStreetMap, our approach shows a significant improvement over not using a map at all. Combined with our method for knowledge distillation, we achieve results that are close to the original HD map-reliant models. Our publicly available navigation map API for Argoverse enables researchers to develop and evaluate their own approaches using navigation maps 4.

IROS Conference 2023 Conference Paper

Joint Out-of-Distribution Detection and Uncertainty Estimation for Trajectory Prediction

Julian Wiederer
Julian Schmidt
Ulrich Kressel
Klaus Dietmayer
Vasileios Belagiannis

Despite the significant research efforts on trajectory prediction for automated driving, limited work exists on assessing the prediction reliability. To address this limitation we propose an approach that covers two sources of error, namely novel situations with out-of-distribution (OOD) detection and the complexity in in-distribution (ID) situations with uncertainty estimation. We introduce two modules next to an encoder-decoder network for trajectory prediction. Firstly, a Gaussian mixture model learns the probability density function of the ID encoder features during training, and then it is used to detect the OOD samples in regions of the feature space with low likelihood. Secondly, an error regression network is applied to the encoder, which learns to estimate the trajectory prediction error in supervised training. During inference, the estimated prediction error is used as the uncertainty. In our experiments, the combination of both modules outperforms the prior work in OOD detection and uncertainty estimation, on the Shifts robust trajectory prediction dataset by 2. 8 % and 10. 1%, respectively. The code is publicly available 4 4 project page: https://github.com/againerju/joodu.

ICRA Conference 2023 Conference Paper

Tackling Clutter in Radar Data - Label Generation and Detection Using PointNet++

Johannes Kopp
Dominik Kellner
Aldi Piroli
Klaus Dietmayer

Radar sensors employed for environment perception, e. g. in autonomous vehicles, output a lot of unwanted clutter. These points, for which no corresponding real objects exist, are a major source of errors in following processing steps like object detection or tracking. We therefore present two novel neural network setups for identifying clutter. The input data, network architectures and training configuration are adjusted specifically for this task. Special attention is paid to the downsampling of point clouds composed of multiple sensor scans. In an extensive evaluation, the new setups display substantially better performance than existing approaches. Because there is no suitable public data set in which clutter is annotated, we design a method to automatically generate the respective labels. By applying it to existing data with object annotations and releasing its code, we effectively create the first freely available radar clutter data set representing realworld driving scenarios. Code and instructions are accessible at www.github.com/kopp-j/clutter-ds.

ICRA Conference 2022 Conference Paper

CRAT-Pred: Vehicle Trajectory Prediction with Crystal Graph Convolutional Neural Networks and Multi-Head Self-Attention

Julian Schmidt
Julian Jordan
Franz Gritschneder
Klaus Dietmayer

Predicting the motion of surrounding vehicles is essential for autonomous vehicles, as it governs their own motion plan. Current state-of-the-art vehicle prediction models heavily rely on map information. In reality, however, this information is not always available. We therefore propose CRAT-Pred, a multi-modal and non-rasterization-based trajectory prediction model, specifically designed to effectively model social interactions between vehicles, without relying on map information. CRAT-Pred applies a graph convolution method originating from the field of material science to vehicle prediction, allowing to efficiently leverage edge features, and combines it with multi-head self-attention. Compared to other map-free approaches, the model achieves state-of-the-art performance with a significantly lower number of model parameters. In addition to that, we quantitatively show that the self-attention mechanism is able to learn social interactions between vehicles, with the weights representing a measurable interaction score. The source code is publicly available 3 3 Source code: https://github.com/schmidt-ju/crat-pred.

IJCAI Conference 2022 Conference Paper

MotionMixer: MLP-based 3D Human Body Pose Forecasting

Arij Bouazizi
Adrian Holzbock
Ulrich Kressel
Klaus Dietmayer
Vasileios Belagiannis

In this work, we present MotionMixer, an efficient 3D human body pose forecasting model based solely on multi-layer perceptrons (MLPs). MotionMixer learns the spatial-temporal 3D body pose dependencies by sequentially mixing both modalities. Given a stacked sequence of 3D body poses, a spatial-MLP extracts fine-grained spatial dependencies of the body joints. The interaction of the body joints over time is then modelled by a temporal MLP. The spatial-temporal mixed features are finally aggregated and decoded to obtain the future motion. To calibrate the influence of each time step in the pose sequence, we make use of squeeze-and-excitation (SE) blocks. We evaluate our approach on Human3. 6M, AMASS, and 3DPW datasets using the standard evaluation protocols. For all evaluations, we demonstrate state-of-the-art performance, while having a model with a smaller number of parameters. Our code is available at: https: //github. com/MotionMLP/MotionMixer.

PDF Details DOI

ICRA Conference 2021 Conference Paper

Dynamic Occupancy Grid Mapping with Recurrent Neural Networks

Marcel Schreiber
Vasileios Belagiannis
Claudius Gläser
Klaus Dietmayer

Modeling and understanding the environment is an essential task for autonomous driving. In addition to the detection of objects, in complex traffic scenarios the motion of other road participants is of special interest. Therefore, we propose to use a recurrent neural network to predict a dynamic occupancy grid map, which divides the vehicle surrounding in cells, each containing the occupancy probability and a velocity estimate. During training, our network is fed with sequences of measurement grid maps, which encode the lidar measurements of a single time step. Due to the combination of convolutional and recurrent layers, our approach is capable to use spatial and temporal information for the robust detection of static and dynamic environment. In order to apply our approach with measurements from a moving ego-vehicle, we propose a method for ego-motion compensation that is applicable in neural network architectures with recurrent layers working on different resolutions. In our evaluations, we compare our approach with a state-of-the-art particle-based algorithm on a large publicly available dataset to demonstrate the improved accuracy of velocity estimates and the more robust separation of the environment in static and dynamic area. Additionally, we show that our proposed method for ego-motion compensation leads to comparable results in scenarios with stationary and with moving ego-vehicle.

IROS Conference 2021 Conference Paper

The Radar Ghost Dataset - An Evaluation of Ghost Objects in Automotive Radar Data

Florian Kraus
Nicolas Scheiner
Werner Ritter
Klaus Dietmayer

Radar sensors have a long tradition in advanced driver assistance systems (ADAS) and also play a major role in current concepts for autonomous vehicles. Their importance is reasoned by their high robustness against meteorological effects, such as rain, snow, or fog, and the radar’s ability to measure relative radial velocity differences via the Doppler effect. The cause for these advantages, namely the large wavelength, is also one of the drawbacks of radar sensors. Compared to camera or lidar sensor, a lot more surfaces in a typical traffic scenario appear flat relative to the radar’s emitted signal. This results in multi-path reflections or so called ghost detections in the radar signal. Ghost objects pose a major source for potential false positive detections in a vehicle’s perception pipeline. Therefore, it is important to be able to segregate multipath reflections from direct ones. In this article, we present a dataset with detailed manual annotations for different kinds of ghost detections. Moreover, two different approaches for identifying these kinds of objects are evaluated. We hope that our dataset encourages more researchers to engage in the fields of multi-path object suppression or exploitation.

IROS Conference 2020 Conference Paper

Inferring Spatial Uncertainty in Object Detection

Zining Wang
Di Feng
Yiyang Zhou
Lars Rosenbaum
Fabian Timm
Klaus Dietmayer
Masayoshi Tomizuka
Wei Zhan

The availability of real-world datasets is the prerequisite for developing object detection methods for autonomous driving. While ambiguity exists in object labels due to error-prone annotation process or sensor observation noises, current object detection datasets only provide deterministic annotations without considering their uncertainty. This precludes an in-depth evaluation among different object detection methods, especially for those that explicitly model predictive probability. In this work, we propose a generative model to estimate bounding box label uncertainties from LiDAR point clouds, and define a new representation of the probabilistic bounding box through spatial distribution. Comprehensive experiments show that the proposed model represents uncertainties commonly seen in driving scenarios. Based on the spatial distribution, we further propose an extension of IoU, called the Jaccard IoU (JIoU), as a new evaluation metric that incorporates label uncertainty. Experiments on the KITTI and the Waymo Open Datasets show that JIoU is superior to IoU when evaluating probabilistic object detectors.

ICRA Conference 2020 Conference Paper

Motion Estimation in Occupancy Grid Maps in Stationary Settings Using Recurrent Neural Networks

Marcel Schreiber
Vasileios Belagiannis
Claudius Gläser
Klaus Dietmayer

In this work, we tackle the problem of modeling the vehicle environment as dynamic occupancy grid map in complex urban scenarios using recurrent neural networks. Dynamic occupancy grid maps represent the scene in a bird's eye view, where each grid cell contains the occupancy prob-ability and the two dimensional velocity. As input data, our approach relies on measurement grid maps, which contain occupancy probabilities, generated with lidar measurements. Given this configuration, we propose a recurrent neural net-work architecture to predict a dynamic occupancy grid map, i. e. filtered occupancy and velocity of each cell, by using a sequence of measurement grid maps. Our network architecture contains convolutional long-short term memories in order to sequentially process the input, makes use of spatial context, and captures motion. In the evaluation, we quantify improvements in estimating the velocity of braking and turning vehicles compared to the state-of-the-art. Additionally, we demonstrate that our approach provides more consistent velocity estimates for dynamic objects, as well as, less erroneous velocity estimates in static area.

ICRA Conference 2019 Conference Paper

Long-Term Occupancy Grid Prediction Using Recurrent Neural Networks

Marcel Schreiber
Stefan Hoermann 0002
Klaus Dietmayer

We tackle the long-term prediction of scene evolution in a complex downtown scenario for automated driving based on Lidar grid fusion and recurrent neural networks (RNNs). A bird's eye view of the scene, including occupancy and velocity, is fed as a sequence to a RNN which is trained to predict future occupancy. The nature of prediction allows generation of multiple hours of training data without the need of manual labeling. Thus, the training strategy and loss function are designed for long sequences of real-world data (unbalanced, continuously changing situations, false labels, etc.). The deep CNN architecture comprises convolutional long short-term memories (ConvLSTMs) to separate static from dynamic regions and to predict dynamic objects in future frames. Novel recurrent skip connections show the ability to predict small occluded objects, i. e. pedestrians, and occluded static regions. Spatio-temporal correlations between grid cells are exploited to predict multimodal future paths and interactions between objects. Experiments also quantity improvements to our previous network, a Monte Carlo approach, and literature.

IROS Conference 2018 Conference Paper

Disparity Sliding Window: Object Proposals from Disparity Images

Julian Müller 0001
Andreas Fregin
Klaus Dietmayer

Sliding window approaches have been widely used for object recognition tasks in recent years [19], [4], [5], [18]. They guarantee an investigation of the entire input image for the object to be detected and allow a localization of that object. Despite the current trend towards deep neural networks, sliding window methods are still used in combination with convolutional neural networks [22]. The risk of overlooking an object is clearly reduced compared to alternative detection approaches which detect objects based on shape, edges or color. Nevertheless, the sliding window technique strongly increases the computational effort as the classifier has to verify a large number of object candidates. This paper proposes a sliding window approach which also uses depth information from a stereo camera. This leads to a greatly decreased number of object candidates without significantly reducing the detection accuracy. A theoretical investigation of the conventional sliding window approach is presented first. Other publications to date only mentioned rough estimations of the computational cost. A mathematical derivation clarifies the number of object candidates with respect to parameters such as image and object size. Subsequently, the proposed disparity sliding window approach is presented in detail. The approach is evaluated on pedestrian detection with annotations and images from the KITTI [10] object detection benchmark. Furthermore, a comparison with two state-of-the-art methods is made. Code is available in C++ and Python https://github.com/julimueller/disparity-sliding-window.

ICRA Conference 2018 Conference Paper

Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labeling

Stefan Hoermann 0002
Martin Bach
Klaus Dietmayer

Long-term situation prediction plays a crucial role for intelligent vehicles. A major challenge still to overcome is the prediction of complex downtown scenarios with multiple road users, e. g. , pedestrians, bikes, and motor vehicles, interacting with each other. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation, and machine learning as long-term predictor. More specifically, a dynamic occupancy grid map is utilized as input to a deep convolutional neural network. This yields the advantage of using spatially distributed velocity estimates from a single time step for prediction, rather than a raw data sequence, alleviating common problems dealing with input time series of multiple sensors. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. Pixel-wise balancing is applied in the loss function counteracting the extreme imbalance between static and dynamic cells. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data and compared to Monte-Carlo simulation. Experiments show the ability to model complex interactions.

IROS Conference 2018 Conference Paper

Fast Trajectory Planning for Automated Vehicles Using Gradient-Based Nonlinear Model Predictive Control

Franz Gritschneder
Knut Graichen
Klaus Dietmayer

Motion trajectory planning is one crucial aspect for automated vehicles, as it governs the own future behavior in a dynamically changing environment. A good utilization of a vehicle's characteristics requires the consideration of the nonlinear system dynamics within the optimization problem to be solved. In particular, real-time feasibility is essential for automated driving, in order to account for the fast changing surrounding, e. g. for moving objects. The key contributions of this paper are the presentation of a fast optimization algorithm for trajectory planning including the nonlinear system model. Further, a new concurrent operation scheme for two optimization algorithms is derived and investigated. The proposed algorithm operates in the submillisecond range on a standard PC. As an exemplary scenario, the task of driving along a challenging reference course is demonstrated.

ICRA Conference 2018 Conference Paper

The DriveU Traffic Light Dataset: Introduction and Comparison with Existing Datasets

Andreas Fregin
Julian Müller 0001
Ulrich Krebel
Klaus Dietmayer

Autonomous driving is a topic in computer vision which has captured a great deal of attention in recent years. One key problem is the detection and state analysis of traffic lights. Even over time, very few datasets for research in this topic have been published and they vary widely in quantity and in quality. To address the complexity of traffic light recognition, we introduce the DriveU**driveU is a joint innovation center of the Daimler AG and the University of Ulm Traffic Light Dataset (DTLD), a large-scale dataset consisting of more than 230, 000 annotations. All annotations are hand-labeled according to strict rules and show a high quality. Recordings were made in eleven different cities during different weather conditions. Our dataset exceeds previous traffic light datasets in size, variance, annotation quality and amount of additional sensor data. We prove the extent of our dataset by an extensive comparison with existing traffic light datasets. Miscellaneous dataset criteria are compared, illustrated and statistically analyzed. In the process, metrics to express the quality and variance of datasets are developed and verified. The dataset can be downloaded from http://traffic-light-data.de.

ICRA Conference 2017 Conference Paper

Vehicle tracking using extended object methods: An approach for fusing radar and laser

Alexander Scheel
Stephan Reuter
Klaus Dietmayer

Combining data from heterogeneous sensors allows to enhance tracking systems by increasing the field of view, incorporating redundancy, and improving the performance by exploiting complementary sensor characteristics. This paper proposes a new vehicle tracking approach for vehicle environment perception that fuses radar and laser data. A Random-Finite-Set-based tracking filter, which permits a clear mathematical formulation of the multi-object problem, is used as fusion center. In combination with extended object measurement models that work on the raw sensor data directly, the filter uses all available information without the need for further preprocessing routines, considers object interdependencies, and works in ambiguous situations. The results are evaluated using experimental data from a test vehicle.

ICRA Conference 2014 Conference Paper

Instantaneous ego-motion estimation using multiple Doppler radars

Dominik Kellner
Michael Barjenbruch
Jens Klappstein
Jürgen Dickmann
Klaus Dietmayer

The estimation of the ego-vehicle's motion is a key capability for advanced driving assistant systems and mobile robot localization. The following paper presents a robust algorithm using radar sensors to instantly determine the complete 2D motion state of the ego-vehicle (longitudinal, lateral velocity and yaw rate). It evaluates the relative motion between at least two Doppler radar sensors and their received stationary reflections (targets). Based on the distribution of their radial velocities across the azimuth angle, non-stationary targets and clutter are excluded. The ego-motion and its corresponding covariance matrix are estimated. The algorithm does not require any preprocessing steps such as clustering or clutter suppression and does not contain any model assumptions. The sensors can be mounted at any position on the vehicle. A common field of view is not required, avoiding target association in space. As an additional benefit, all targets are instantly labeled as stationary or non-stationary.

ICRA Conference 2013 Conference Paper

Multi-sensor fusion with out-of-sequence measurements for vehicle environment perception

Antje Westenberger
Steffen Wäldele
Balaganesh Dora
Bharanidhar Duraisamy
Marc M. Muntzinger
Klaus Dietmayer

Automated driving applications require an environment perception that is reliable and fast. Multi-sensor fusion is a suitable means to combine the advantages of different measurement principles. However, this may lead to out-of-sequence measurements, i. e. , asynchronous measurements where the original order of the measurements is lost. High-performance out-of-sequence algorithms are therefore needed that do not depend on the order of the measurements. In addition, existence probabilities can increase the reliability of the fusion system especially in safety critical applications. This paper presents a novel approach to handle out-of-sequence measurements not only in state estimation, but also in existence estimation. The method is shown to result in equal or less computational costs than state-of-the-art methods. The proposed algorithm is evaluated with real world data from crash tests.

ICRA Conference 2010 Conference Paper

Consistent mapping of multistory buildings by introducing global constraints to graph-based SLAM

Michael Karg
Kai M. Wurm
Cyrill Stachniss
Klaus Dietmayer
Wolfram Burgard

In the past, there has been a tremendous advance in the area of simultaneous localization and mapping (SLAM). However, there are relatively few approaches for incorporating prior information or knowledge about structural similarities into the mapping process. Consider, for example, office buildings in which most of the offices have an identical geometric layout. The same typically holds for the individual stories of buildings. In this paper, we propose an approach for generating alignment constraints between different floors of the same building in the context of graph-based SLAM. This is done under the assumption that the individual floors of a building share at least some structural properties. To identify such areas, we apply a particle filter-based localization approach using maps and observations from different floors. We evaluate our system using several real datasets as well as in simulation. The results demonstrate that our approach is able to correctly align multiple floors and allows the robot to generate consistent models of multi-story buildings.