Arjun Majumdar

ICML Conference 2025 Conference Paper

LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D

• Paul McVay
• Sergio Arnaud
• Ada Martin
• Arjun Majumdar
• Krishna Murthy Jatavallabhula
• Phillip Thomas
• Ruslan Partsey
• Daniel Dugas

We present LOCATE 3D, a model for localizing objects in 3D scenes from referring expressions like "the small coffee table between the sofa and the lamp. " LOCATE 3D sets a new state-of-the-art on standard referential grounding benchmarks and showcases robust generalization capabilities. Notably, LOCATE 3D operates directly on sensor observation streams (posed RGB-D frames), enabling real-world deployment on robots and AR devices. Key to our approach is 3D-JEPA, a novel self-supervised learning (SSL) algorithm applicable to sensor point clouds. It takes as input a 3D pointcloud featurized using 2D foundation models (CLIP, DINO). Subsequently, masked prediction in latent space is employed as a pretext task to aid the self-supervised learning of contextualized pointcloud features. Once trained, the 3D-JEPA encoder is finetuned alongside a language-conditioned decoder to jointly predict 3D masks and bounding boxes. Additionally, we introduce LOCATE 3D DATASET, a new dataset for 3D referential grounding, spanning multiple capture setups with over 130K annotations. This enables a systematic study of generalization capabilities as well as a stronger model. Code, models and dataset can be found at the project website: locate3d. atmeta. com

IROS Conference 2024 Conference Paper

Embodiment Randomization for Cross Embodiment Navigation

• Pranav Putta
• Gunjan Aggarwal
• Roozbeh Mottaghi
• Dhruv Batra
• Naoki Yokoyama
• Joanne Truong
• Arjun Majumdar

We present Embodiment Randomization, a simple, inexpensive, and intuitive technique for training robust behavior policies that can be transferred to multiple robot embodiments. While prior works require real-world data from multiple robots, or complex algorithmic adjustments to address the challenge of embodiment generalization, our approach leverages the power of simulation and large-scale reinforcement learning and can be easily integrated within existing policy learning methods. We show that policies trained with embodiment randomization implicitly perform system identification, enabling them to adapt to new embodiments during deployment. Our approach not only shows significant improvements in adapting to novel robot configurations, but also in generalizing from simulation to reality and contending with real-world perturbations, highlighting the potential of embodiment randomization in creating versatile and adaptable robotic navigation policies.

ICRA Conference 2024 Conference Paper

Stretch with Stretch: Physical Therapy Exercise Games Led by a Mobile Manipulator

• Matthew Lamsey
• You Liang Tan
• Meredith D. Wells
• Madeline Beatty
• Zexuan Liu
• Arjun Majumdar
• Kendra Washington
• Jerome Feldman

Physical therapy (PT) is a key component of many rehabilitation regimens, such as treatments for Parkinson’s disease (PD). However, there are shortages of physical therapists and adherence to self-guided PT is low. Robots have the potential to support physical therapists and increase adherence to self-guided PT, but prior robotic systems have been large and immobile, which can be a barrier to use in homes and clinics. We present Stretch with Stretch (SWS), a novel robotic system for leading stretching exercise games for older adults with PD. SWS consists of a compact and lightweight mobile manipulator (Hello Robot Stretch RE1) that visually and verbally guides users through PT exercises. The robot’s soft end effector serves as a target that users repetitively reach towards and press with a hand, foot, or knee. For each exercise, target locations are customized for the individual via a visually estimated kinematic model, a haptically estimated range of motion, and the person’s exercise performance. The system includes sound effects and verbal feedback from the robot to keep users engaged throughout a session and augment physical exercise with cognitive exercise. We conducted a user study for which people with PD (n = 10) performed 6 exercises with the system. Participants perceived the SWS to be useful and easy to use. They also reported mild to moderate perceived exertion (RPE).

NeurIPS Conference 2024 Conference Paper

Subsurface Scattering for Gaussian Splatting

• Jan-Niklas Dihlmann
• Arjun Majumdar
• Andreas Engelhardt
• Raphael Braun
• Hendrik P. Lensch

3D reconstruction and relighting of objects made from scattering materials present a significant challenge due to the complex light transport beneath the surface. 3D Gaussian Splatting introduced high-quality novel view synthesis at real-time speeds. While 3D Gaussians efficiently approximate an object's surface, they fail to capture the volumetric properties of subsurface scattering. We propose a framework for optimizing an object's shape together with the radiance transfer field given multi-view OLAT (one light at a time) data. Our method decomposes the scene into an explicit surface represented as 3D Gaussians, with a spatially varying BRDF, and an implicit volumetric representation of the scattering component. A learned incident light field accounts for shadowing. We optimize all parameters jointly via ray-traced differentiable rendering. Our approach enables material editing, relighting, and novel view synthesis at interactive rates. We show successful application on synthetic data and contribute a newly acquired multi-view multi-light dataset of objects in a light-stage setup. Compared to previous work we achieve comparable or better results at a fraction of optimization and rendering time while enabling detailed control over material attributes.

ICRA Conference 2024 Conference Paper

What Do We Learn from a Large-Scale Study of Pre-Trained Visual Representations in Sim and Real Environments?

• Sneha Silwal
• Karmesh Yadav
• Tingfan Wu
• Jay Vakil
• Arjun Majumdar
• Sergio Arnaud
• Claire Chen
• Vincent-Pierre Berges

We present a large empirical investigation on the use of pre-trained visual representations (PVRs) for training downstream policies that execute real-world tasks. Our study involves five different PVRs, each trained for five distinct manipulation or indoor navigation tasks. We performed this evaluation using three different robots and two different policy learning paradigms. From this e ort, we can arrive at three insights: 1) the performance trends of PVRs in the simulation are generally indicative of their trends in the real world, 2) the use of PVRs enables a first-of-its-kind result with indoor ImageNav (zero-shot transfer to a held-out scene in the real world), and 3) the benefits from variations in PVRs, primarily data-augmentation and fine-tuning, also transfer to the real-world performance. See project website 1 for additional details and visuals.

ICML Conference 2023 Conference Paper

Masked Trajectory Models for Prediction, Representation, and Control

• Philipp Wu
• Arjun Majumdar
• Kevin Stone
• Yixin Lin
• Igor Mordatch
• Pieter Abbeel
• Aravind Rajeswaran

We introduce Masked Trajectory Models (MTM) as a generic abstraction for sequential decision making. MTM takes a trajectory, such as a state-action sequence, and aims to reconstruct the trajectory conditioned on random subsets of the same trajectory. By training with a highly randomized masking pattern, MTM learns versatile networks that can take on different roles or capabilities, by simply choosing appropriate masks at inference time. For example, the same MTM network can be used as a forward dynamics model, inverse dynamics model, or even an offline RL agent. Through extensive experiments in several continuous control tasks, we show that the same MTM network – i. e. same weights – can match or outperform specialized networks trained for the aforementioned capabilities. Additionally, we find that state representations learned by MTM can significantly accelerate the learning speed of traditional RL algorithms. Finally, in offline RL benchmarks, we find that MTM is competitive with specialized offline RL algorithms, despite MTM being a generic self-supervised learning method without any explicit RL components. Code is available at https: //github. com/facebookresearch/mtm.

NeurIPS Conference 2023 Conference Paper

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

• Arjun Majumdar
• Karmesh Yadav
• Sergio Arnaud
• Jason Ma
• Claire Chen
• Sneha Silwal
• Aryan Jain
• Vincent-Pierre Berges

We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual ‘foundation models’ for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4, 000 hours of egocentric videos from 7 different sources (over 4. 3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10, 000 GPU-hours to train) for the benefit of the research community.

ICML Conference 2022 Conference Paper

SSL Enables Learning from Sparse Rewards in Image-Goal Navigation

• Arjun Majumdar
• Gunnar A. Sigurdsson
• Robinson Piramuthu
• Jesse Thomason
• Dhruv Batra
• Gaurav S. Sukhatme

NeurIPS Conference 2022 Conference Paper

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

• Arjun Majumdar
• Gunjan Aggarwal
• Bhavika Devnani
• Judy Hoffman
• Dhruv Batra

We present a scalable approach for learning open-world object-goal navigation (ObjectNav) – the task of asking a virtual robot (agent) to find any instance of an object in an unexplored environment (e. g. , “find a sink”). Our approach is entirely zero-shot – i. e. , it does not require ObjectNav rewards or demonstrations of any kind. Instead, we train on the image-goal navigation (ImageNav) task, in which agents find the location where a picture (i. e. , goal image) was captured. Specifically, we encode goal images into a multimodal, semantic embedding space to enable training semantic-goal navigation (SemanticNav) agents at scale in unannotated 3D environments (e. g. , HM3D). After training, SemanticNav agents can be instructed to find objects described in free-form natural language (e. g. , “sink, ” “bathroom sink, ” etc. ) by projecting language goals into the same multimodal, semantic embedding space. As a result, our approach enables open-world ObjectNav. We extensively evaluate our agents on three ObjectNav datasets (Gibson, HM3D, and MP3D) and observe absolute improvements in success of 4. 2% - 20. 0% over existing zero-shot methods. For reference, these gains are similar or better than the 5% improvement in success between the Habitat 2020 and 2021 ObjectNav challenge winners. In an open-world setting, we discover that our agents can generalize to compound instructions with a room explicitly mentioned (e. g. , “Find a kitchen sink”) and when the target room can be inferred (e. g. , “Find a sink and a stove”).

NeurIPS Conference 2021 Conference Paper

SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation

• Abhinav Moudgil
• Arjun Majumdar
• Harsh Agrawal
• Stefan Lee
• Dhruv Batra

Natural language instructions for visual navigation often use scene descriptions (e. g. , bedroom) and object references (e. g. , green chairs) to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining (i. e. , learning the alignment between images and text from large-scale web data) to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1. 8% absolute in SPL on R2R and 3. 7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.