Author name cluster

Katie Luo

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

IROS Conference 2025 Conference Paper

Enhanced Motion Forecasting with Plug-and-Play Multimodal Large Language Models

Katie Luo
Jingwei Ji
Tong He
Runsheng Xu
Yichen Xie 0002
Dragomir Anguelov
Mingxing Tan

Current autonomous driving systems rely on specialized models for perceiving and predicting motion, which demonstrate reliable performance in standard conditions. However, generalizing cost-effectively to diverse real-world scenarios remains a significant challenge. To address this, we propose Plug-and-Forecast (PnF), a plug-and-play approach that augments existing motion forecasting models with multimodal large language models (MLLMs). PnF builds on the insight that natural language provides a more effective way to describe and handle complex scenarios, enabling quick adaptation to targeted behaviors. We design prompts to extract structured scene understanding from MLLMs and distill this information into learnable embeddings to augment existing behavior prediction models. Our method leverages the zero-shot reasoning capabilities of MLLMs to achieve significant improvements in motion prediction performance, while requiring no fine-tuning—making it practical to adopt. We validate our approach on two state-of-the-art motion forecasting models using the Waymo Open Motion Dataset and the nuScenes Dataset, demonstrating consistent performance improvements across both benchmarks.

Details

ICRA Conference 2024 Conference Paper

Augmenting Lane Perception and Topology Understanding with Standard Definition Navigation Maps

Katie Luo
Xinshuo Weng
Yan Wang 0051
Shuang Wu
Jie Li
Kilian Q. Weinberger
Yue Wang 0041
Marco Pavone 0001

Autonomous driving has traditionally relied heavily on costly and labor-intensive High Definition (HD) maps, hindering scalability. In contrast, Standard Definition (SD) maps are more affordable and have worldwide coverage, offering a scalable alternative. In this work, we systematically explore the effect of SD maps for real-time lane-topology understanding. We propose a novel framework to integrate SD maps into online map prediction and propose a Transformer-based encoder, SD Map Encoder Representations from transFormers, to leverage priors in SD maps for the lane-topology prediction task. This enhancement consistently and significantly boosts (by up to 60%) lane detection and topology prediction on current state-of-the-art online map prediction methods without bells and whistles and can be immediately incorporated into any Transformer-based lane-topology method. Code is available at https://github.com/NVlabs/SMERF.

Details

ICRA Conference 2024 Conference Paper

Better Monocular 3D Detectors with LiDAR from the Past

Yurong You
Cheng Perng Phoo
Carlos Andres Diaz-Ruiz
Katie Luo
Wei-Lun Chao
Mark E. Campbell
Bharath Hariharan
Kilian Q. Weinberger

Accurate 3D object detection is crucial to autonomous driving. Though LiDAR-based detectors have achieved impressive performance, the high cost of LiDAR sensors precludes their widespread adoption in affordable vehicles. Camera-based detectors are cheaper alternatives but often suffer inferior performance compared to their LiDAR-based counterparts due to inherent depth ambiguities in images. In this work, we seek to improve monocular 3D detectors by leveraging unlabeled historical LiDAR data. Specifically, at inference time, we assume that the camera-based detectors have access to multiple unlabeled LiDAR scans from past traversals at locations of interest (potentially from other high-end vehicles equipped with LiDAR sensors). Under this setup, we proposed a novel, simple, and end-to-end trainable framework, termed AsyncDepth, to effectively extract relevant features from asynchronous LiDAR traversals of the same location for monocular 3D detectors. We show consistent and significant performance gain (up to 9 AP) across multiple state-of-the-art models and datasets with a negligible additional latency of 9. 66 ms and a small storage cost. Our code can be found at https://github.com/YurongYou/AsyncDepth.

Details

ICLR Conference 2024 Conference Paper

Pre-training LiDAR-based 3D Object Detectors through Colorization

Tai-Yu Pan
Chenyang Ma
Tianle Chen
Cheng Perng Phoo
Katie Luo
Yurong You
Mark E. Campbell
Kilian Q. Weinberger

Accurate 3D object detection and understanding for self-driving cars heavily relies on LiDAR point clouds, necessitating large amounts of labeled data to train. In this work, we introduce an innovative pre-training approach, Grounded Point Colorization (GPC), to bridge the gap between data and labels by teaching the model to colorize LiDAR point clouds, equipping it with valuable semantic cues. To tackle challenges arising from color variations and selection bias, we incorporate color as "context" by providing ground-truth colors as hints during colorization. Experimental results on the KITTI and Waymo datasets demonstrate GPC's remarkable effectiveness. Even with limited labeled data, GPC significantly improves fine-tuning performance; notably, on just 20% of the KITTI dataset, GPC outperforms training from scratch with the entire dataset. In sum, we introduce a fresh perspective on pre-training for 3D object detection, aligning the objective with the model's intended role and ultimately advancing the accuracy and efficiency of 3D object detection for autonomous vehicles.

Details

NeurIPS Conference 2023 Conference Paper

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

Katie Luo
Zhenzhen Liu
Xiangyu Chen
Yurong You
Sagie Benaim
Cheng Perng Phoo
Mark Campbell
Wen Sun

Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles—where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i. e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i. e. , boxes containing objects are scored higher than those without. We start from the detector’s own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https: //github. com/katieluo88/DRIFT.

PDF Details

ICLR Conference 2022 Conference Paper

Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception

Yurong You
Katie Luo
Xiangyu Chen 0007
Junan Chen
Wei-Lun Chao
Wen Sun 0002
Bharath Hariharan
Mark E. Campbell

Self-driving cars must detect vehicles, pedestrians, and other trafﬁc participants accurately to operate safely. Small, far-away, or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them. To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the same scene. We posit that these past data, which are typically discarded, provide rich contextual information for disambiguating the above-mentioned challenging cases. To this end, we propose a novel end-to-end trainable Hindsight framework to extract this contextual information from past traversals and store it in an easy-to-query data structure, which can then be leveraged to aid future 3D object detection of the same scene. We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision on multiple autonomous driving datasets, most notably by more than 300% on the challenging cases. Our code is available at https://github.com/YurongYou/Hindsight.

Details

NeurIPS Conference 2022 Conference Paper

Unsupervised Adaptation from Repeated Traversals for Autonomous Driving

Yurong You
Cheng Perng Phoo
Katie Luo
Travis Zhang
Wei-Lun Chao
Bharath Hariharan
Mark Campbell
Kilian Q. Weinberger

For a self-driving car to operate reliably, its perceptual system must generalize to the end-user's environment --- ideally without additional annotation efforts. One potential solution is to leverage unlabeled data (e. g. , unlabeled LiDAR point clouds) collected from the end-users' environments (i. e. target domain) to adapt the system to the difference between training and testing environments. While extensive research has been done on such an unsupervised domain adaptation problem, one fundamental problem lingers: there is no reliable signal in the target domain to supervise the adaptation process. To overcome this issue we observe that it is easy to collect unsupervised data from multiple traversals of repeated routes. While different from conventional unsupervised domain adaptation, this assumption is extremely realistic since many drivers share the same roads. We show that this simple additional assumption is sufficient to obtain a potent signal that allows us to perform iterative self-training of 3D object detectors on the target domain. Concretely, we generate pseudo-labels with the out-of-domain detector but reduce false positives by removing detections of supposedly mobile objects that are persistent across traversals. Further, we reduce false negatives by encouraging predictions in regions that are not persistent. We experiment with our approach on two large-scale driving datasets and show remarkable improvement in 3D object detection of cars, pedestrians, and cyclists, bringing us a step closer to generalizable autonomous driving.

PDF Details

IROS Conference 2021 Conference Paper

Safety-Oriented Pedestrian Occupancy Forecasting

Katie Luo
Sergio Casas 0002
Renjie Liao 0001
Xinchen Yan
Yuwen Xiong
Wenyuan Zeng
Raquel Urtasun

In this paper we address an important problem in self-driving, forecasting multi-pedestrian motion and their shared scene occupancy map, which is critical for safe navigation. Our contributions are two-fold. First, we advocate for predicting both the individual motions as well as the scene occupancy map in order to effectively deal with missing detections caused by postprocessing, e. g. confidence thresholding and non-maximum suppression. Second, we propose a Scene-Actor Graph Neural Network (SA-GNN) which captures the interactions among pedestrians within the same scene, including those that have not been detected, by preserving the relative spatial information of pedestrians via 2D convolution and via message passing. We show that our scene-occupancy predictions are more accurate than those from state-of-the-art motion forecasting methods, while also matching their performance in pedestrian motion forecasting metrics on two large-scale real-world datasets, nuScenes and ATG4D.

Details