Michael S. Ryoo

ICLR Conference 2025 Conference Paper

LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

• Xiang Li 0109
• Cristina Mata
• Jongwoo Park 0003
• Kumara Kahatapitiya
• Yoo Sung Jang
• Jinghuan Shang
• Kanchana Ranasinghe
• Ryan D. Burgert

Vision Language Models (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language-Action (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by a limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets, aligning robotic actions with image pixel coordinates. Further, we enhance this dataset in a self-supervised manner by defining six auxiliary tasks, without requiring any additional action annotations. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across multiple simulated and real-world tasks, we demonstrate that LLaRA achieves state-of-the-art performance while preserving the generalization capabilities of large language models. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA.

ICLR Conference 2025 Conference Paper

Understanding Long Videos with Multimodal Language Models

• Kanchana Ranasinghe
• Xiang Li 0109
• Kumara Kahatapitiya
• Michael S. Ryoo

Large Language Models (LLMs) have allowed recent LLM-based approaches to achieve excellent performance on long-video understanding benchmarks. We investigate how extensive world knowledge and strong reasoning skills of underlying LLMs influence this strong performance. Surprisingly, we discover that LLM-based approaches can yield surprisingly good accuracy on long-video tasks with limited video information, sometimes even with no video-specific information. Building on this, we explore injecting video-specific information into an LLM-based framework. We utilize off-the-shelf vision tools to extract three object-centric information modalities from videos, and then leverage natural language as a medium for fusing this information. Our resulting Multimodal Video Understanding (MVU) framework demonstrates state-of-the-art performance across multiple video understanding benchmarks. Strong performance also on robotics domain tasks establishes its strong generality. Code: github.com/kahnchana/mvu

ICRA Conference 2024 Conference Paper

Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning

• Xiang Li 0109
• Varun Belagali
• Jinghuan Shang
• Michael S. Ryoo

Diffusion models have been adopted for behavioral cloning in a sequence modeling fashion, benefiting from their exceptional capabilities in modeling complex data distributions. The standard diffusion-based policy iteratively denoises action sequences from random noise conditioned on the input states and the model is typically trained with a singular diffusion loss. This paper explores the potential enhancements in such models when the denoising process is informed by a better visual representation. We study the scenario where the model is jointly optimized using the standard diffusion loss alongside an auxiliary objective based on self-supervised learning. After experimenting with various objectives, we introduce Crossway Diffusion, a simple yet effective way to enhance diffusion-based visuomotor policy learning via a state decoder and an auxiliary reconstruction objective. During training, the state decoder reconstructs raw image pixels and other states from the intermediate representations of the model. Experiments demonstrate the effectiveness of our method in various simulated and real-world tasks, confirming its consistent advantages over the standard diffusion-based policy and other baselines.

ICRA Conference 2024 Conference Paper

SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attention

• Isabel Leal
• Krzysztof Choromanski
• Deepali Jain
• Avinava Dubey
• Jake Varley
• Michael S. Ryoo
• Yao Lu 0006
• Frederick Liu

We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment. SARA-RT relies on the new method of fine-tuning proposed by us, called up-training. It converts pre-trained or already fine-tuned Transformer-based robotic policies of quadratic time complexity (including massive billion-parameter vision-language-action models or VLAs), into their efficient linear-attention counterparts maintaining high quality. We demonstrate the effectiveness of SARA-RT by speeding up: (a) the class of recently introduced RT-2 models [1], the first VLA robotic policies pre-trained on internet-scale data, as well as (b) Point Cloud Transformer (PCT) robotic policies operating on large point clouds. We complement our results with the rigorous mathematical analysis providing deeper insight into the phenomenon of SARA.

ICRA Conference 2023 Conference Paper

Energy-Based Models for Cross-Modal Localization using Convolutional Transformers

• Alan Wu
• Michael S. Ryoo

We present a novel framework using Energy-Based Models (EBMs) for localizing a ground vehicle mounted with a range sensor against satellite imagery in the absence of GPS. Lidar sensors have become ubiquitous on autonomous vehicles for describing its surrounding environment. Map priors are typically built using the same sensor modality for localization purposes. However, these map building endeavors using range sensors are often expensive and time-consuming. Alternatively, we leverage the use of satellite images as map priors, which are widely available, easily accessible, and pro-vide comprehensive coverage. We propose a method using convolutional transformers that performs accurate metric-level localization in a cross-modal manner, which is challenging due to the drastic difference in appearance between the sparse range sensor readings and the rich satellite imagery. We train our model end-to-end and demonstrate our approach achieving higher accuracy than the state-of-the-art on KITTI, Pandaset, and a custom dataset.

ICRA Conference 2023 Conference Paper

Open-vocabulary Queryable Scene Representations for Real World Planning

• Boyuan Chen 0003
• Fei Xia 0002
• Brian Ichter
• Kanishka Rao
• Keerthana Gopalakrishnan
• Michael S. Ryoo
• Austin Stone
• Daniel Kappler

Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable scene representation with Visual Language models (VLMs). An LLM based object proposal module parses instructions and proposes involved objects to query the scene representation for object availability and location. An LLM planner then plans with such information about the scene. NLMap allows robots to operate without a fixed list of objects nor executable options, enabling real robot operation unachievable by previous methods. Project website: https://nlmap-saycan.github.io.

ICLR Conference 2023 Conference Paper

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

• Andy Zeng 0001
• Maria Attarian
• Brian Ichter
• Krzysztof Choromanski
• Adrian Wong
• Stefan Welker
• Federico Tombari
• Aveek Purohit

We investigate how multimodal prompt engineering can use language as the intermediate representation to combine complementary knowledge from different pretrained (potentially multimodal) language models for a variety of tasks. This approach is both distinct from and complementary to the dominant paradigm of joint multimodal training. It also recalls a traditional systems-building view as in classical NLP pipelines, but with prompting large pretrained multimodal models. We refer to these as Socratic Models (SMs): a modular class of systems in which multiple pretrained models may be composed zero-shot via multimodal-informed prompting to capture new multimodal capabilities, without additional finetuning. We show that these systems provide competitive state-of-the-art performance for zero-shot image captioning and video-to-text retrieval, and also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes), and (iii) robot perception and planning. We hope this work provides (a) results for stronger zero-shot baseline performance with analysis also highlighting their limitations, (b) new perspectives for building multimodal systems powered by large pretrained models, and (c) practical application advantages in certain regimes limited by data scarcity, training compute, or model access.

IJCAI Conference 2023 Conference Paper

SWAT: Spatial Structure Within and Among Tokens

• Kumara Kahatapitiya
• Michael S. Ryoo

Modeling visual data as tokens (i. e. , image patches) using attention mechanisms, feed-forward networks or convolutions has been highly effective in recent years. Such methods usually have a common pipeline: a tokenization method, followed by a set of layers/blocks for information mixing, both within and among tokens. When image patches are converted into tokens, they are often flattened, discarding the spatial structure within each patch. As a result, any processing that follows (eg: multi-head self-attention) may fail to recover and/or benefit from such information. In this paper, we argue that models can have significant gains when spatial structure is preserved during tokenization, and is explicitly used during the mixing stage. We propose two key contributions: (1) Structure-aware Tokenization and, (2) Structure-aware Mixing, both of which can be combined with existing models with minimal effort. We introduce a family of models (SWAT), showing improvements over the likes of DeiT, MLP-Mixer and Swin Transformer, across multiple benchmarks including ImageNet classification and ADE20K segmentation. Our code is available at github. com/kkahatapitiya/SWAT.

AAAI Conference 2023 Conference Paper

Weakly-Guided Self-Supervised Pretraining for Temporal Activity Detection

• Kumara Kahatapitiya
• Zhou Ren
• Haoxiang Li
• Zhenyu Wu
• Michael S. Ryoo
• Gang Hua

Temporal Activity Detection aims to predict activity classes per frame, in contrast to video-level predictions in Activity Classification (i.e., Activity Recognition). Due to the expensive frame-level annotations required for detection, the scale of detection datasets is limited. Thus, commonly, previous work on temporal activity detection resorts to fine-tuning a classification model pretrained on large-scale classification datasets (e.g., Kinetics-400). However, such pretrained models are not ideal for downstream detection, due to the disparity between the pretraining and the downstream fine-tuning tasks. In this work, we propose a novel weakly-guided self-supervised pretraining method for detection. We leverage weak labels (classification) to introduce a self-supervised pretext task (detection) by generating frame-level pseudo labels, multi-action frames, and action segments. Simply put, we design a detection task similar to downstream, on large-scale classification data, without extra annotations. We show that the models pretrained with the proposed weakly-guided self-supervised detection task outperform prior work on multiple challenging activity detection benchmarks, including Charades and MultiTHUMOS. Our extensive ablations further provide insights on when and how to use the proposed models for activity detection. Code is available at github.com/kkahatapitiya/SSDet.

ICLR Conference 2022 Conference Paper

Hybrid Random Features

• Krzysztof Choromanski
• Han Lin
• Haoxian Chen 0002
• Arijit Sehanobish
• Yuanzhe Ma
• Deepali Jain
• Jake Varley
• Andy Zeng 0001

We propose a new class of random feature methods for linearizing softmax and Gaussian kernels called hybrid random features (HRFs) that automatically adapt the quality of kernel estimation to provide most accurate approximation in the defined regions of interest. Special instantiations of HRFs lead to well-known methods such as trigonometric (Rahimi & Recht, 2007) or (recently introduced in the context of linear-attention Transformers) positive random features (Choromanski et al., 2021). By generalizing Bochner’s Theorem for softmax/Gaussian kernels and leveraging random features for compositional kernels, the HRF-mechanism provides strong theoretical guarantees - unbiased approximation and strictly smaller worst-case relative errors than its counterparts. We conduct exhaustive empirical evaluation of HRF ranging from pointwise kernel estimation experiments, through tests on data admitting clustering structure to benchmarking implicit-attention Transformers (also for downstream Robotics applications), demonstrating its quality in a wide spectrum of machine learning problems.

IROS Conference 2021 Conference Paper

Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning

• Jinghuan Shang
• Michael S. Ryoo

Humans learn to imitate by observing others. However, robot imitation learning generally requires expert demonstrations in the first-person view (FPV). Collecting such FPV videos for every robot could be very expensive. Third-person imitation learning (TPIL) is the concept of learning action policies by observing other agents in a third-person view (TPV), similar to what humans do. This ultimately allows utilizing human and robot demonstration videos in TPV from many different data sources, for the policy learning. In this paper, we present a TPIL approach for robot tasks with egomotion. Although many robot tasks with ground/aerial mobility often involve actions with camera egomotion, study on TPIL for such tasks has been limited. Here, FPV and TPV observations are visually very different; FPV shows egomotion while the agent appearance is only observable in TPV. To enable better state learning for TPIL, we propose our disentangled representation learning method. We use a dual auto-encoder structure plus representation permutation loss and time-contrastive loss to ensure the state and viewpoint representations are well disentangled. Our experiments show the effectiveness of our approach.

ICRA Conference 2021 Conference Paper

Visionary: Vision architecture discovery for robot learning

• Iretiayo Akinola
• Anelia Angelova
• Yao Lu 0006
• Yevgen Chebotar
• Dmitry Kalashnikov
• Jacob Varley
• Julian Ibarz
• Michael S. Ryoo

We propose a vision-based architecture search algorithm for robot manipulation learning, which discovers interactions between low dimension action inputs and high dimensional visual inputs. Our approach automatically designs architectures while training on the task – discovering novel ways of combining and attending image feature representations with actions as well as features from previous layers. The obtained new architectures demonstrate better task success rates, in some cases with a large margin, compared to a recent high performing baseline. Our real robot experiments also confirm that it improves grasping performance by 6%. This is the first approach to demonstrate a successful neural architecture search and attention connectivity search for a real-robot task.

ICLR Conference 2020 Conference Paper

AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures

• Michael S. Ryoo
• A. J. Piergiovanni
• Mingxing Tan
• Anelia Angelova

Learning to represent videos is a very challenging task both algorithmically and computationally. Standard video CNN architectures have been designed by directly extending architectures devised for image understanding to include the time dimension, using modules such as 3D convolutions, or by using two-stream design to capture both appearance and motion in videos. We interpret a video CNN as a collection of multi-stream convolutional blocks connected to each other, and propose the approach of automatically finding neural architectures with better connectivity and spatio-temporal interactions for video understanding. This is done by evolving a population of overly-connected architectures guided by connection weight learning. Architectures combining representations that abstract different input types (i.e., RGB and optical flow) at multiple temporal resolutions are searched for, allowing different types or sources of information to interact with each other. Our method, referred to as AssembleNet, outperforms prior approaches on public video datasets, in some cases by a great margin. We obtain 58.6% mAP on Charades and 34.27% accuracy on Moments-in-Time.

AAAI Conference 2020 Conference Paper

Differentiable Grammars for Videos

• AJ Piergiovanni
• Anelia Angelova
• Michael S. Ryoo

This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos. Learning latent terminals, non-terminals, and production rules directly from continuous data allows the construction of a generative model capturing sequential structures with multiple possibilities. Our model is fully differentiable, and provides easily interpretable results which are important in order to understand the learned structures. It outperforms the state-of-the-art on several challenging datasets and is more accurate for forecasting future activities in videos. We plan to open-source the code.1

IROS Conference 2019 Conference Paper

Learning Real-World Robot Policies by Dreaming

• A. J. Piergiovanni
• Alan Wu
• Michael S. Ryoo

Learning to control robots directly based on images is a primary challenge in robotics. However, many existing reinforcement learning approaches require iteratively obtaining millions of robot samples to learn a policy, which can take significant time. In this paper, we focus on learning a realistic world model capturing the dynamics of scene changes conditioned on robot actions. Our dreaming model can emulate samples equivalent to a sequence of images from the actual environment, technically by learning an action-conditioned future representation/scene regressor. This allows the agent to learn action policies (i. e. , visuomotor policies) by interacting with the dreaming model rather than the real-world. We experimentally confirm that our dreaming model enables robot learning of policies that transfer to the real-world.

IROS Conference 2019 Conference Paper

Privacy-Preserving Robot Vision with Anonymized Faces by Extreme Low Resolution

• Myeung Un Kim
• Harim Lee
• Hyun Jong Yang
• Michael S. Ryoo

As smart cameras are becoming ubiquitous in mobile robot systems, there is an increasing concern in camera devices invading people’s privacy by recording unwanted images. We want to fundamentally protect privacy by blurring unwanted blocks in images, such as faces, yet ensure that the robots can understand the video for their perception. In this paper, we propose a novel mobile robot framework with a deep learning-based privacy-preserving camera system. The proposed camera system detects privacy-sensitive blocks, i. e. , human face, from extreme low resolution (LR) images, and then dynamically enhances the resolution of only privacy-insensitive blocks, e. g. , backgrounds. Keeping all the face blocks to be extreme LR of 15x15 pixels, we can guarantee that human faces are never at high resolution (HR) in any of processing or memory, thus yielding strong privacy protection even from cracking or backdoors. Our camera system produces an image on a real-time basis, the human faces of which are in extreme LR while the backgrounds are in HR. We experimentally confirm that our proposed face detection camera system outperforms the state-of-the-art small face detection algorithm, while the robot performs ORB-SLAM2 well even with videos of extreme LR faces. Therefore, with the proposed system, we do not too much sacrifice robot perception performance to protect privacy.

ICML Conference 2019 Conference Paper

Temporal Gaussian Mixture Layer for Videos

• A. J. Piergiovanni
• Michael S. Ryoo

We introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer and present how it can be used to efficiently capture longer-term temporal information in continuous activity videos. The TGM layer is a temporal convolutional layer governed by a much smaller set of parameters (e. g. , location/variance of Gaussians) that are fully differentiable. We present our fully convolutional video models with multiple TGM layers for activity detection. The extensive experiments on multiple datasets, including Charades and MultiTHUMOS, confirm the effectiveness of TGM layers, significantly outperforming the state-of-the-arts.

IROS Conference 2017 Conference Paper

Learning robot activities from first-person human videos using convolutional future regression

• Jangwon Lee 0002
• Michael S. Ryoo

We design a new approach that allows robot learning of new activities from unlabeled human example videos. Given videos of humans executing the same activity from a human's viewpoint (i. e. , first-person videos), our objective is to make the robot learn the temporal structure of the activity as its future regression network, and learn to transfer such model for its own motor execution. We present a new deep learning model: We extend the state-of-the-art convolutional object detection network for the representation/estimation of human hands in training videos, and newly introduce the concept of using a fully convolutional network to regress (i. e. , predict) the intermediate scene representation corresponding to the future frame (e. g. , 1-2 seconds later). Combining these allows direct prediction of future locations of human hands and objects, which enables the robot to infer the motor control plan using our manipulation network. We experimentally confirm that our approach makes learning of robot activities from unlabeled human interaction videos possible, and demonstrate that our robot is able to execute the learned collaborative activities in real-time directly based on its camera input.

ICRA Conference 2017 Conference Paper

Learning social affordance grammar from videos: Transferring human interactions to human-robot interactions

• Tianmin Shu
• Xiaofeng Gao 0002
• Michael S. Ryoo
• Song-Chun Zhu

In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI). Based on Gibbs sampling, our weakly supervised grammar learning can automatically construct a hierarchical representation of an interaction with long-term joint sub-tasks of both agents and short term atomic actions of individual agents. Based on a new RGB-D video dataset with rich instances of human interactions, our experiments of Baxter simulation, human evaluation, and real Baxter test demonstrate that the model learned from limited training data successfully generates human-like behaviors in unseen scenarios and outperforms both baselines.

IROS Conference 2012 Conference Paper

Reliable object detection and segmentation using inpainting

• Ji Hoon Joung
• Michael S. Ryoo
• Sunglok Choi
• Sung-Rak Kim

This paper presents a novel object detection and segmentation method utilizing an inpainting algorithm. Inpainting is a concept of recovering missing image regions based on their surroundings, which were originally used for restoration of damaged paintings. In this paper, we newly utilize inpainting to judge whether an object candidate region includes the foreground object or not. The key idea is that if we erase a certain region from an image, the inpainting algorithm is expected to recover the erased image only when it belongs a background area (i. e. only when there is no object in it). By measuring the similarity between the inpainted region and the original image region, our approach filters out false detections while maintaining true object detections. Furthermore, we take advantage of the inpainting for object segmentation, since our approach is designed to explicitly distinguish foreground areas from its background. Experimental results confirm that our approach applied to baseline detectors enables better recognition of objects, obtaining higher accuracies. We illustrate how our inpainting-based detection/segmentation approach benefits the object detection using two different pedestrian datasets.

IJCAI Conference 2007 Conference Paper

• Michael S. Ryoo
• J. K. Aggarwal

We introduce a human-computer interaction system which collaborates with a user by providing feedback during user activities. The goal of the system is to help a user complete a high-level activity that has been represented hierarchically. While a user is performing the high-level activity, our system analyzes what sub-events a user has already completed and what sub-events are needed next in order for the user to finish the activity. The representations of human activities are constructed using a previously developed context-free grammar based representation scheme. We focus on a game named 'pentagram game' to illustrate our system. In the experiments, our system shows the ability to guide the user to complete the 'pentagram game' by providing explicit feedback. The feedback not only contains atomic level instructions but also describes higher-level long-term goals of the composite activities.