Arrow Research search

Author name cluster

Aniket Bera

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

40 papers
2 author rows

Possible papers

40

TMLR Journal 2026 Journal Article

Learning Lagrangian Interaction Dynamics with Sampling-Based Model Order Reduction

  • Hrishikesh Viswanath
  • Yue Chang
  • Aleksey Panas
  • Julius Berner
  • Peter Yichen Chen
  • Aniket Bera

Simulating physical systems governed by Lagrangian dynamics often entails solving partial differential equations (PDEs) over high-resolution spatial domains, leading to significant computational expense. Reduced-order modeling (ROM) mitigates this cost by evolving low-dimensional latent representations of the underlying system. While neural ROMs enable querying solutions from latent states at arbitrary spatial points, their latent states typically represent the global domain and struggle to capture localized, highly dynamic behaviors such as fluids. We propose a sampling-based reduction framework that evolves Lagrangian systems directly in physical space, over the particles themselves, reducing the number of active degrees of freedom via data-driven neural PDE operators. To enable querying at arbitrary spatial locations, we introduce a learnable kernel parameterization that uses local spatial information from time-evolved sample particles to infer the underlying solution manifold. Empirically, our approach achieves a 6.6$\times$–32$\times$ reduction in input dimensionality while maintaining high-fidelity evaluations across diverse Lagrangian regimes, including fluid flows, granular media, and elastoplastic dynamics. We refer to this framework as GIOROM (\textbf{G}eometry-\textbf{I}nf\textbf{O}rmed \textbf{R}educed-\textbf{O}rder \textbf{M}odeling). All of our code and data is available at \url{https://github.com/HrishikeshVish/GIOROM}

ICRA Conference 2025 Conference Paper

COLLAGE: Collaborative Human-Agent Interaction Generation Using Hierarchical Latent Diffusion and Language Models

  • Divyanshu Daiya
  • Damon Conover
  • Aniket Bera

We propose a novel framework COLLAGE for generating collaborative agent-object-agent interactions by leveraging large language models (LLMs) and hierarchical motion-specific vector-quantized variational autoencoders (VQ-VAEs). Our model addresses the lack of rich datasets in this domain by incorporating the knowledge and reasoning abilities of LLMs to guide a generative diffusion model. The hierarchical VQ-VAE architecture captures different motion-specific characteristics at multiple levels of abstraction, avoiding redundant concepts and enabling efficient multi-resolution representation. We introduce a diffusion model that operates in the latent space and incorporates LLM-generated motion planning cues to guide the denoising process, resulting in prompt-specific motion generation with greater control and diversity. Experimental results on the CORE-4D, and InterHuman datasets demonstrate the effectiveness of our approach in generating realistic and diverse collaborative human-object-human interactions, outperforming state-of-the-art methods. Our work opens up new possibilities for modeling complex interactions in various domains, such as robotics, graphics and computer vision. Paper website: https://collagemotion.github.io/

IROS Conference 2025 Conference Paper

Dynamic Obstacle Avoidance through Uncertainty-Based Adaptive Planning with Diffusion

  • Vineet Punyamoorty
  • Pascal Jutras-Dubé
  • Ruqi Zhang
  • Vaneet Aggarwal
  • Damon Conover
  • Aniket Bera

By framing reinforcement learning as a sequence modeling problem, recent work has enabled the use of generative models, such as diffusion models, for planning. While these models are effective in predicting long-horizon state trajectories in deterministic environments, they face challenges in dynamic settings with moving obstacles. Effective collision avoidance demands continuous monitoring and adaptive decision-making. While re-planning at every time step could ensure safety, it introduces substantial computational overhead due to the repetitive prediction of overlapping state sequences—a process that is particularly costly with diffusion models, known for their intensive iterative sampling procedure. We propose an adaptive generative planning approach that dynamically adjusts re-planning frequency based on the uncertainty of action predictions. Our method minimizes the need for frequent, computationally expensive, and redundant re-planning while maintaining robust collision avoidance performance. In experiments, we obtain a 13. 5% increase in the mean trajectory length and 12. 7% increase in mean reward over long-horizon planning, indicating a reduction in collision rates, and improved ability to navigate the environment safely.

IROS Conference 2025 Conference Paper

EASEIR: Efficient and Adaptive Safe-set Estimation via Implicit Representation for High-dimensional Motion Planning

  • Hojun Lee
  • Yuseop Sim
  • Changheon Han
  • Jiho Lee
  • Aniket Bera
  • Changju Kim
  • Martin Byung-Guk Jun

Collision-free robotic manipulation is extremely important for all safety-critical applications of robots. Especially for large-scale automation in modern manufacturing facilities where numerous hardware and software systems collaborate in relatively structured environments, accomplishing effectiveness, efficiency, and safety in not only repetitive tasks but also their sporadic reconfigurations is ideal. Yet, existing online and offline Motion Planning (MP) algorithms do not meet such a unique combination of harsh demands, since most of the advances in MP aim for a subset of the requirements. To bridge the gap, we introduce a novel implicit neural function (EASEIR) designed for efficient offline safe set composition for robotic manipulators operating in structured environments. Addressing the challenges of managing high-dimensional configuration spaces (C-space), EASEIR leverages Implicit Neural Representations (INR) to relate coordinates of a discretized robot operation space with collision sets in C-space. EASEIR then utilizes the mapping to actively compose a collision-free set in response to arbitrary occupancy of the operation space by obstacles. The proposed method comprises three core modules: (a) Latent Key Generator (LKG) that maps the coordinates of the space to intermediate latent keys, (b) Latent Key Decoder (LKD) that reconstructs collision sets from the keys, and (c) Full Set Compositor (FSC) that generates a full collision-free set using set operations. On a 6 Degrees of Freedom (DoF) arm, EASEIR generates safe configuration sets nearly 43 times faster than the state-of-the-art analytical method while maintaining comparable accuracy (∼ 0. 2% collision) during evaluations in a simulation environment.

IROS Conference 2025 Conference Paper

EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answering

  • Kai Cheng
  • Zhengyuan Li
  • Xingpeng Sun
  • Byung-Cheol Min
  • Amrit Singh Bedi
  • Aniket Bera

Embodied Question Answering (EQA) is an essential yet challenging task for robot assistants. Large vision-language models (VLMs) have shown promise for EQA, but existing approaches either treat it as static video question answering without active exploration or restrict answers to a closed set of choices. These limitations hinder real-world applicability, where a robot must explore efficiently and provide accurate answers in open-vocabulary settings. To overcome these challenges, we introduce EfficientEQA, a novel framework that couples efficient exploration with free-form answer generation. EfficientEQA features three key innovations: (1) Semantic-Value-Weighted Frontier Exploration (SFE) with Verbalized Confidence (VC) from a black-box VLM to prioritize semantically important areas to explore, enabling the agent to gather relevant information faster; (2) a BLIP relevancy-based mechanism to stop adaptively by flagging highly relevant observations as outliers to indicate whether the agent has collected enough information; and (3) a Retrieval-Augmented Generation (RAG) method for the VLM to answer accurately based on pertinent images from the agent’s observation history without relying on predefined choices. Our experimental results show that EfficientEQA achieves over 15% higher answer accuracy and requires over 20% fewer exploration steps than state-of-the-art methods. Our code is available at: https://github.com/chengkaiAcademyCity/EfficientEQA

IROS Conference 2025 Conference Paper

Go-SLAM: Grounded Object Segmentation and Localization with Gaussian Splatting SLAM

  • Phu Pham
  • Dipam Patel
  • Damon Conover
  • Aniket Bera

We introduce Go-Slam, a novel framework that combines 3D Gaussian Splatting SLAM with grounded object segmentation and open-vocabulary querying to enable object-aware 3D scene reconstruction. Go-Slam incrementally builds high-fidelity 3D maps from RGB-D inputs while embedding semantic information by assigning unique object identifiers to Gaussian primitives. This integration allows the system to support flexible, natural language queries and accurately localize objects in complex, static environments. To achieve robust semantic mapping, Go-Slam leverages object detection and segmentation models, enabling consistent object identification across frames without relying on predefined categories. We evaluate Go-Slam across diverse indoor scenes, demonstrating improvements over existing baselines in both reconstruction quality and object localization accuracy. Our results show that Go-Slam effectively bridges the gap between geometric mapping and semantic understanding, supporting real-time scene interaction and object retrieval in open-world environments.

ICRA Conference 2025 Conference Paper

HEROES: Unreal Engine-based Human and Emergency Robot Operation Education System

  • Anav Chaudhary
  • Kshitij Tiwari
  • Aniket Bera

Training and preparing first responders and humanitarian robots for Mass Casualty Incidents (MCIs) often poses a challenge owing to the lack of realistic and easily accessible test facilities. While such facilities can offer realistic scenarios post an MCI that can serve training and educational purposes for first responders and humanitarian robots, they are often hard to access owing to logistical constraints. To overcome this challenge, we present HEROES- a versatile Unreal Engine-based simulator for designing novel training simulations for humans and emergency robots for such urban search and rescue operations. The proposed HEROES simulator is capable of generating synthetic datasets for machine learning pipelines that are used for training robot navigation. This work addresses the necessity for a comprehensive training platform in the robotics community, ensuring pragmatic and efficient preparation for real-world emergency scenarios. The strengths of our simulator lie in its adaptability, scalability, and ability to facilitate collaboration between robot developers and first responders, fostering synergy in developing effective strategies for search and rescue operations in MCIs. We conducted a preliminary user study with an average score of 8. 1 out of 10 supporting the ability of HEROES to generate sufficiently varied environments and a score of 7. 8 out of 10 affirming the usefulness of the simulation environment. HEROES has been integrated with ROS and has been used to train an RL model for a real robot as a proof of concept.

ICRA Conference 2025 Conference Paper

Hypergraph-Based Coordinated Task Allocation and Socially-Aware Navigation for Multi-Robot Systems

  • Weizheng Wang 0004
  • Aniket Bera
  • Byung-Cheol Min

A team of multiple robots seamlessly and safely working in human-filled public environments requires adaptive task allocation and socially-aware navigation that account for dynamic human behavior. Current approaches struggle with highly dynamic pedestrian movement and the need for flexible task allocation. We propose Hyper-SAMARL, a hypergraph-based system for multi-robot task allocation and socially-aware navigation, leveraging multi-agent reinforcement learning (MARL). Hyper-SAMARL models the environmental dynamics between robots, humans, and points of interest (POIs) using a hypergraph, enabling adaptive task assignment and socially-compliant navigation through a hypergraph diffusion mechanism. Our framework, trained with MARL, effectively captures interactions between robots and humans, adapting tasks based on real-time changes in human activity. Experimental results demonstrate that Hyper-SAMARL outperforms baseline models in terms of social navigation, task completion efficiency, and adaptability in various simulated scenarios 1 1 The experimental videos and additional information about this work can be found at: https://sites.google.com/view/hyper-samarl. .

IROS Conference 2025 Conference Paper

PathCluster: Pedestrian Group-Adaptive Social Navigation in Dense Crowds

  • Nihal Gunukula
  • Aniket Bera

Mobile robot navigation in crowded spaces is crucial for deployment but remains challenging in extremely dense environments. While recent works have utilized predictive measures to anticipate individual trajectories, bettering overall navigation, these methods often fail to scale in high-density crowds due to computational intensity. To mitigate this issue we propose PathCluster, a novel approach that generates groups based on individuals with similar trajectories to enable efficient navigation in dense crowds. Our method introduces a group generator algorithm that identifies and treats clusters as cohesive units, significantly decreasing the complexity of trajectory prediction while maintaining its benefits. Simulation results demonstrate that our method, PathCluster, achieves a 45% higher success rate, and a 25% lower collision rate, and can tackle more challenging navigational tasks within a 48hr-time limit compared to previous social navigation models in extremely crowded environments.

ICRA Conference 2025 Conference Paper

SELP: Generating Safe and Efficient Task Plans for Robot Agents with Large Language Models

  • Yi Wu 0023
  • Zikang Xiong
  • Yiran Hu
  • Shreyash S. Iyengar
  • Nan Jiang 0012
  • Aniket Bera
  • Lin Tan 0001
  • Suresh Jagannathan

Despite significant advancements in large language models (LLMs) that enhance robot agents' understanding and execution of natural language (NL) commands, ensuring the agents adhere to user-specified constraints remains challenging, particularly for complex commands and long-horizon tasks. To address this challenge, we present three key insights, equivalence voting, constrained decoding, and domain-specific fine-tuning, which significantly enhance LLM planners' capability in handling complex tasks. Equivalence voting ensures consistency by generating and sampling multiple Linear Temporal Logic (LTL) formulas from NL commands, grouping equivalent LTL formulas, and selecting the majority group of formulas as the final LTL formula. Constrained decoding then uses the generated LTL formula to enforce the autoregressive inference of plans, ensuring the generated plans conform to the LTL. Domain-specific fine-tuning customizes LLMs to produce safe and efficient plans within specific task domains. Our approach, Safe Efficient LLM Planner (SELP), combines these insights to create LLM planners to generate plans adhering to user commands with high confidence. We demonstrate the effective-ness and generalizability of SELP across different robot agents and tasks, including drone navigation and robot manipulation. For drone navigation tasks, SELP outperforms state-of-the-art planners by 10. 8% in safety rate (i. e. , finishing tasks conforming to NL commands) and by 19. 8% in plan efficiency. For robot manipulation tasks, SELP achieves 20. 4% improvement in safety rate. Our datasets for evaluating NL-to-LTL and robot task planning will be released in github.com/lt-asset/selp.

AAAI Conference 2025 Conference Paper

VidSole: A Multimodal Dataset for Joint Kinetics Quantification and Disease Detection with Deep Learning

  • Archit Kambhamettu
  • Samantha Snyder
  • Maliheh Fakhar
  • Samuel Audia
  • Ross Miller
  • Jae Kun Shim
  • Aniket Bera

Understanding internal joint loading is critical for diagnosing gait-related diseases such as knee osteoarthritis; however, current methods of measuring joint risk factors are time-consuming, expensive, and restricted to lab settings. In this paper, we enable the large-scale, cost-effective biomechanical analysis of joint loading via three key contributions: the development and deployment of novel instrumented insoles, the creation of a large multimodal biomechanics dataset (VidSole), and a baseline deep learning pipeline to predict internal joint loading factors. Our novel instrumented insole measures the tri-axial forces and moments across five high-pressure points under the foot. VidSole consists of the forces and moments measured by these insoles along with corresponding RGB video from two viewpoints, 3D body motion capture, and force plate data for over 2,600 trials of 52 diverse participants performing four fundamental activities of daily living (sit-to-stand, stand-to-sit, walking, and running). We feed the insole data and kinematic parameters extractable from video (i.e., pose, knee angle) into a deep learning pipeline consisting of an ensemble Gated Recurrent Unit (GRU) activity classifier followed by activity-specific Long Short Term Memory (LSTM) regression networks to estimate knee adduction moment (KAM), a biomechanical risk factor for knee osteoarthritis. The successful classification of activities at an accuracy of 99.02 percent and KAM estimation with mean absolute error (MAE) less than 0.5 percent*body weight*height, the current threshold for accurately detecting knee osteoarthritis with KAM, illustrates the usefulness of our dataset for future research and clinical settings.

IROS Conference 2024 Conference Paper

Adaptive Planning with Generative Models under Uncertainty

  • Pascal Jutras-Dubé
  • Ruqi Zhang
  • Aniket Bera

Planning with generative models has emerged as an effective decision-making paradigm across a wide range of domains, including reinforcement learning and autonomous navigation. While continuous replanning at each timestep might seem intuitive because it allows decisions to be made based on the most recent environmental observations, it results in substantial computational challenges, primarily due to the complexity of the generative model’s underlying deep learning architecture. Our work addresses this challenge by introducing a simple adaptive planning policy that leverages the generative model’s ability to predict long-horizon state trajectories, enabling the execution of multiple actions consecutively without the need for immediate replanning. We propose to use the predictive uncertainty derived from a Deep Ensemble of inverse dynamics models to dynamically adjust the intervals between planning sessions. In our experiments conducted on locomotion tasks within the OpenAI Gym framework, we demonstrate that our adaptive planning policy allows for a reduction in replanning frequency to only about 10% of the steps without compromising the performance. Our results underscore the potential of generative modeling as an efficient and effective tool for decision-making.

TMLR Journal 2024 Journal Article

Beyond Text: Utilizing Vocal Cues to Improve Decision Making in LLMs for Robot Navigation Tasks

  • Xingpeng Sun
  • Haoming Meng
  • Souradip Chakraborty
  • Amrit Bedi
  • Aniket Bera

While LLMs excel in processing text in these human conversations, they struggle with the nuances of verbal instructions in scenarios like social navigation, where ambiguity and uncertainty can erode trust in robotic and other AI systems. We can address this shortcoming by moving beyond text and additionally focusing on the paralinguistic features of these audio responses. These features are the aspects of spoken communication that do not involve the literal wording (lexical content) but convey meaning and nuance through how something is said. We present ``Beyond Text''; an approach that improves LLM decision-making by integrating audio transcription along with a subsection of these features, which focus on the affect and more relevant in human-robot conversations. This approach not only achieves a 70.26% winning rate, outperforming existing LLMs by 22.16% to 48.30% (gemini-1.5-pro and gpt-3.5 respectively), but also enhances robustness against token manipulation adversarial attacks, highlighted by a 22.44% less decrease ratio than the text-only language model in winning rate. We also present the first dataset on disfluent human audio-guided instructions for future research in this field. ``Beyond Text'' marks an advancement in social robot navigation and broader Human-Robot interactions, seamlessly integrating text-based guidance with human-audio-informed language models.

AAAI Conference 2024 Conference Paper

DanceAnyWay: Synthesizing Beat-Guided 3D Dances with Randomized Temporal Contrastive Learning

  • Aneesh Bhattacharya
  • Manas Paranjape
  • Uttaran Bhattacharya
  • Aniket Bera

We present DanceAnyWay, a generative learning method to synthesize beat-guided dances of 3D human characters synchronized with music. Our method learns to disentangle the dance movements at the beat frames from the dance movements at all the remaining frames by operating at two hierarchical levels. At the coarser "beat" level, it encodes the rhythm, pitch, and melody information of the input music via dedicated feature representations only at the beat frames. It leverages them to synthesize the beat poses of the target dances using a sequence-to-sequence learning framework. At the finer "repletion" level, our method encodes similar rhythm, pitch, and melody information from all the frames of the input music via dedicated feature representations. It generates the full dance sequences by combining the synthesized beat and repletion poses and enforcing plausibility through an adversarial learning framework. Our training paradigm also enforces fine-grained diversity in the synthesized dances through a randomized temporal contrastive loss, which ensures different segments of the dance sequences have different movements and avoids motion freezing or collapsing to repetitive movements. We evaluate the performance of our approach through extensive experiments on the benchmark AIST++ dataset and observe improvements of about 7%-12% in motion quality metrics and 1.5%-4% in motion diversity metrics over the current baselines, respectively. We also conducted a user study to evaluate the visual quality of our synthesized dances. We noted that, on average, the samples generated by our method were about 9-48% more preferred by the participants and had a 4-27% better five-point Likert-scale score over the best available current baseline in terms of motion quality and synchronization. Our source code and project page are available at https://github.com/aneeshbhattacharya/DanceAnyWay.

IROS Conference 2024 Conference Paper

Optimizing Crowd-Aware Multi-Agent Path Finding through Local Communication with Graph Neural Networks

  • Phu Pham
  • Aniket Bera

Multi-Agent Path Finding (MAPF) in crowded environments presents a challenging problem in motion planning, aiming to find collision-free paths for all agents in the system. MAPF finds a wide range of applications in various domains, including aerial swarms, autonomous warehouse robotics, and self-driving vehicles. Current approaches to MAPF generally fall into two main categories: centralized and decentralized planning. Centralized planning suffers from the curse of dimensionality when the number of agents or states increases and thus does not scale well in large and complex environments. On the other hand, decentralized planning enables agents to engage in real-time path planning within a partially observable environment, demonstrating implicit coordination. However, they suffer from slow convergence and performance degradation in dense environments. In this paper, we introduce CRAMP, a novel crowd-aware decentralized reinforcement learning approach to address this problem by enabling efficient local communication among agents via Graph Neural Networks (GNNs), facilitating situational awareness and decision-making capabilities in congested environments. We test CRAMP on simulated environments and demonstrate that our method outperforms the state-of-the-art decentralized methods for MAPF on various metrics. CRAMP improves the solution quality up to 59% measured in makespan and collision count, and up to 35% improvement in success rate in comparison to previous methods.

ICRA Conference 2024 Conference Paper

Trajectory Prediction for Robot Navigation using Flow-Guided Markov Neural Operator

  • Rashmi Bhaskara
  • Hrishikesh Viswanath
  • Aniket Bera

Predicting pedestrian movements remains a complex and persistent challenge in robot navigation research. We must evaluate several factors to achieve accurate predictions, such as pedestrian interactions, the environment, crowd density, and social and cultural norms. Accurate prediction of pedestrian paths is vital for ensuring safe human-robot interaction, especially in robot navigation. Furthermore, this research has potential applications in autonomous vehicles, pedestrian tracking, and human-robot collaboration. Therefore, in this paper, we introduce FlowMNO, an Optical Flow-Integrated Markov Neural Operator designed to capture pedestrian behavior across diverse scenarios. Our paper models trajectory prediction as a Markovian process, where future pedestrian coordinates depend solely on the current state. This problem formulation eliminates the need to store previous states. We conducted experiments using standard benchmark datasets like ETH, HOTEL, ZARA1, ZARA2, UCY, and RGB-D pedestrian datasets. Our study demonstrates that FlowMNO outperforms some of the state-of-the-art deep learning methods like LSTM, GAN, and CNN-based approaches, by approximately 86. 46% when predicting pedestrian trajectories. Thus, we show that FlowMNO can seamlessly integrate into robot navigation systems, enhancing their ability to navigate crowded areas smoothly.

IROS Conference 2024 Conference Paper

TrustNavGPT: Modeling Uncertainty to Improve Trustworthiness of Audio-Guided LLM-Based Robot Navigation

  • Xingpeng Sun
  • Yiran Zhang
  • Xindi Tang
  • Amrit Singh Bedi
  • Aniket Bera

Large language models (LLMs) exhibit a wide range of promising capabilities – from step-by-step planning to commonsense reasoning –that provide utility for robot navigation. However, as humans communicate with robots in the real world, ambiguity and uncertainty may be embedded inside spoken instructions. While LLMs are proficient at processing text in human conversations, they often encounter difficulties with the nuances of verbal instructions and, thus, remain prone to hallucinate trust in human command. In this work, we present TrustNavGPT, an LLM-based audio-guided navigation agent that uses affective cues in spoken communication—elements such as tone and inflection that convey meaning beyond words—allowing it to assess the trustworthiness of human commands and make effective, safe decisions. Experiments across a variety of simulation and real-world setups show a 70. 46% success rate in catching command uncertainty and an 80% success rate in finding the target, 48. 30%, and 55% outperform existing LLM-based navigation methods, respectively. Additionally, TrustNavGPT shows remarkable resilience against adversarial attacks, highlighted by a 22%+ less decrease ratio than the existing LLM navigation method in success rate. Our approach provides a lightweight yet effective approach that extends existing LLMs to model audio vocal features embedded in the voice command and model uncertainty for safe robotic navigation. For more information, visit the TrustNav project page.

ICRA Conference 2023 Conference Paper

AZTR: Aerial Video Action Recognition with Auto Zoom and Temporal Reasoning

  • Xijun Wang 0002
  • Ruiqi Xian
  • Tianrui Guan
  • Celso M. de Melo
  • Stephen M. Nogar
  • Aniket Bera
  • Dinesh Manocha

We propose a novel approach for aerial video action recognition. Our method is designed for videos captured using UAVs and can run on edge or mobile devices. We present a learning-based approach that uses customized auto zoom to automatically identify the human target and scale it appropriately. This makes it easier to extract the key features and reduces the computational overhead. We also present an efficient temporal reasoning algorithm to capture the action information along the spatial and temporal domains within a controllable computational cost. Our approach has been implemented and evaluated both on the desktop with high-end GPUs and on the low power Robotics RB5 Platform for robots and drones. In practice, we achieve $6. 1-7. 4 \%$ improvement over SOTA in Top-1 accuracy on the RoCoG-v2 dataset, 8. 3-10. 4% improvement on the UAV-Human dataset and 3. 2% improvement on the Drone Action dataset.

IROS Conference 2023 Conference Paper

DroNeRF: Real-Time Multi-Agent Drone Pose Optimization for Computing Neural Radiance Fields

  • Dipam Patel
  • Phu Pham
  • Aniket Bera

We present a novel optimization algorithm called DroNeRF for the autonomous positioning of monocular camera drones around an object for real-time 3D reconstruction using only a few images. Neural Radiance Fields, or NeRF, is a novel view synthesis technique used to generate new views of an object or scene from a set of input images. Using drones in conjunction with NeRF provides a unique and dynamic way to generate novel views of a scene, especially with limited scene capabilities of restricted movements. Our approach focuses on calculating optimized pose for individual drones while solely depending on the object geometry without using any external localization system. The unique camera positioning during the data capturing phase significantly impacts the quality of the 3D model. To evaluate the quality of our generated novel views, we compute different perceptual metrics like the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). Our work demonstrates the benefit of using an optimal placement of various drones with limited mobility to generate perceptually better results.

ICRA Conference 2023 Conference Paper

EWareNet: Emotion-Aware Pedestrian Intent Prediction and Adaptive Spatial Profile Fusion for Social Robot Navigation

  • Venkatraman Narayanan
  • Bala Murali Manoghar
  • Rama Prashanth RV
  • Aniket Bera

We present EWareNet, a novel intent and affect-aware social robot navigation algorithm among pedestrians. Our approach predicts the trajectory-based pedestrian intent from gait sequence, which is then used for intent-guided navigation taking into account social and proxemic constraints. We propose a transformer-based model that works on commodity RGB-D cameras mounted onto a moving robot. Our intent prediction routine is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. Our navigation scheme consists of a novel obstacle profile representation methodology that is dynamically adjusted based on the pedestrian pose, intent, and affect. The navigation scheme is based on a reinforcement learning algorithm that takes pedestrian intent and robot's impact on pedestrian intent into consideration, in addition to the environmental configuration. We outperform current state-of-art algorithms for intent prediction from 3D gaits.

IROS Conference 2023 Conference Paper

RAIST: Learning Risk Aware Traffic Interactions via Spatio-Temporal Graph Convolutional Networks

  • Videsh Suman
  • Phu Pham
  • Aniket Bera

A key aspect of driving a road vehicle is to interact with other road users, assess their intentions and make riskaware tactical decisions. An intuitive approach to enabling an intelligent automated driving system would be incorporating some aspects of human driving behavior. To this end, we propose a novel driving framework for egocentric views based on spatio-temporal traffic graphs. The traffic graphs model not only the spatial interactions amongst the road users but also their individual intentions through temporally associated message passing. We leverage a spatio-temporal graph convolutional network (ST-GCN) to train the graph edges. These edges are formulated using parameterized functions of 3D positions and scene-aware appearance features of road agents. Along with tactical behavior prediction, it is crucial to evaluate the risk-assessing ability of the proposed framework. We claim that our framework learns risk-aware representations by improving on the task of risk object identification, especially in identifying objects with vulnerable interactions like pedestrians and cyclists.

IROS Conference 2023 Conference Paper

SG-LSTM: Social Group LSTM for Robot Navigation Through Dense Crowds

  • Rashmi Bhaskara
  • Maurice Chiu
  • Aniket Bera

As personal robots become increasingly accessible and affordable, their applications extend beyond large corporate warehouses and factories to operate in diverse, less controlled environments, where they interact with larger groups of people. In such contexts, ensuring not only safety and efficiency but also mitigating potential adverse psychological impacts on humans and adhering to unwritten social norms become paramount. In this research, we aim to address these challenges by developing a cutting-edge model capable of predicting pedestrian movements and interactions in crowded environments. To this end, we propose a novel approach called the Social Group Long Short-term Memory (SG-LSTM) model, which effectively captures the complexities of human group behavior and interactions within dense surroundings. By integrating social awareness into the LSTM architecture, our model achieves significantly enhanced trajectory predictions. The implementation of our SG-LSTM model empowers navigation algorithms to compute collision-free paths faster and with higher accuracy, particularly in complex and crowded scenarios. To foster further advancements in social navigation research, we contribute a substantial video dataset comprising labeled pedestrian groups, which we release to the broader research community. To thoroughly evaluate the performance of our approach, we conduct extensive experiments on multiple datasets, including ETH, Hotel, and MOT15. We compare various prediction approaches, such as LIN, LSTM, O-LSTM, and S-LSTM, and rigorously assess runtime performance.

AAAI Conference 2022 Conference Paper

Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders

  • Abhishek Banerjee
  • Uttaran Bhattacharya
  • Aniket Bera

We present a novel generalized zero-shot algorithm to recognize perceived emotions from gestures. Our task is to map gestures to novel emotion categories not encountered in training. We introduce an adversarial autoencoder-based representation learning that correlates 3D motion-captured gesture sequences with the vectorized representation of the naturallanguage perceived emotion terms using word2vec embeddings. The language-semantic embedding provides a representation of the emotion label space, and we leverage this underlying distribution to map the gesture sequences to the appropriate categorical emotion labels. We train our method using a combination of gestures annotated with known emotion terms and gestures not annotated with any emotions. We evaluate our method on the MPI Emotional Body Expressions Database (EBEDB) and obtain an accuracy of 58. 43%. We see an improvement in performance compared to current state-of-the-art algorithms for generalized zero-shot learning by an absolute 25–27%. We also demonstrate our approach on publicly available online videos and movie scenes, where the actors’ pose has been extracted and mapped to their respective emotive states.

ICRA Conference 2021 Conference Paper

Can a Robot Trust You? : A DRL-Based Approach to Trust-Driven Human-Guided Navigation

  • Vishnu Sashank Dorbala
  • Arjun Srinivasan
  • Aniket Bera

Humans are known to construct cognitive maps of their everyday surroundings using a variety of perceptual inputs. As such, when a human is asked for directions to a particular location, their wayfinding capability in converting this cognitive map into directional instructions is challenged. Owing to spatial anxiety, the language used in the spoken instructions can be vague and often unclear. To account for this unreliability in navigational guidance, we propose a novel Deep Reinforcement Learning (DRL) based trust-driven robot navigation algorithm that learns humans’ trustworthiness to perform a language guided navigation task. Our approach seeks to answer the question as to whether a robot can trust a human’s navigational guidance or not. To this end, we look at training a policy that learns to navigate towards a goal location using only trustworthy human guidance, driven by its own robot trust metric. We look at quantifying various affective features from language-based instructions and incorporate them into our policy’s observation space in the form of a human trust metric. We utilize both these trust metrics into an optimal cognitive reasoning scheme that decides when and when not to trust the given guidance. Our results show that the learned policy can navigate the environment in an optimal, time-efficient manner as opposed to an explorative approach that performs the same task. We showcase the efficacy of our results both in simulation and a real world environment.

IROS Conference 2020 Conference Paper

CMetric: A Driving Behavior Measure using Centrality Functions

  • Rohan Chandra
  • Uttaran Bhattacharya
  • Trisha Mittal
  • Aniket Bera
  • Dinesh Manocha

We present a new measure, CMetric, to classify driver behaviors using centrality functions. Our formulation combines concepts from computational graph theory and social traffic psychology to quantify and classify the behavior of human drivers. CMetric is used to compute the probability of a vehicle executing a driving style, as well as the intensity used to execute the style. Our approach is designed for realtime autonomous driving applications, where the trajectory of each vehicle or road-agent is extracted from a video. We compute a dynamic geometric graph (DGG) based on the positions and proximity of the road-agents and centrality functions corresponding to closeness and degree. These functions are used to compute the CMetric based on style likelihood and style intensity estimates. Our approach is general and makes no assumption about traffic density, heterogeneity, or how driving behaviors change over time. We present an algorithm to compute CMetric and demonstrate its performance on real-world traffic datasets. To test the accuracy of CMetric, we introduce a new evaluation protocol (called "Time Deviation Error") that measures the difference between human prediction and the prediction made by CMetric.

ICRA Conference 2020 Conference Paper

GraphRQI: Classifying Driver Behaviors Using Graph Spectrums

  • Rohan Chandra
  • Uttaran Bhattacharya
  • Trisha Mittal
  • Xiaoyu Li
  • Aniket Bera
  • Dinesh Manocha

We present a novel algorithm (GraphRQI) to identify driver behaviors from road-agent trajectories. Our approach assumes that the road-agents exhibit a range of driving traits, such as aggressive or conservative driving. Moreover, these traits affect the trajectories of nearby road-agents as well as the interactions between road-agents. We represent these inter-agent interactions using unweighted and undirected traffic graphs. Our algorithm classifies the driver behavior using a supervised learning algorithm by reducing the computation to the spectral analysis of the traffic graph. Moreover, we present a novel eigenvalue algorithm to compute the spectrum efficiently. We provide theoretical guarantees for the running time complexity of our eigenvalue algorithm and show that it is faster than previous methods by 2 times. We evaluate the classification accuracy of our approach on traffic videos and autonomous driving datasets corresponding to urban traffic. In practice, GraphRQI achieves an accuracy improvement of up to 25% over prior driver behavior classification algorithms. We also use our classification algorithm to predict the future trajectories of road-agents.

AAAI Conference 2020 Conference Paper

M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues

  • Trisha Mittal
  • Uttaran Bhattacharya
  • Rohan Chandra
  • Aniket Bera
  • Dinesh Manocha

We present M3ER, a learning-based method for emotion recognition from multiple input modalities. Our approach combines cues from multiple co-occurring modalities (such as face, text, and speech) and also is more robust than other methods to sensor noise in any of the individual modalities. M3ER models a novel, data-driven multiplicative fusion method to combine the modalities, which learn to emphasize the more reliable cues and suppress others on a persample basis. By introducing a check step which uses Canonical Correlational Analysis to differentiate between ineffective and effective modalities, M3ER is robust to sensor noise. M3ER also generates proxy features in place of the ineffectual modalities. We demonstrate the efficiency of our network through experimentation on two benchmark datasets, IEMOCAP and CMU-MOSEI. We report a mean accuracy of 82. 7% on IEMOCAP and 89. 0% on CMU-MOSEI, which, collectively, is an improvement of about 5% over prior work.

IROS Conference 2020 Conference Paper

ProxEmo: Gait-based Emotion Learning and Multi-view Proxemic Fusion for Socially-Aware Robot Navigation

  • Venkatraman Narayanan
  • Bala Murali Manoghar
  • Vishnu Sashank Dorbala
  • Dinesh Manocha
  • Aniket Bera

We present ProxEmo, a novel end-to-end emotion prediction algorithm for socially aware robot navigation among pedestrians. Our approach predicts the perceived emotions of a pedestrian from walking gaits, which is then used for emotion-guided navigation taking into account social and proxemic constraints. To classify emotions, we propose a multi-view skeleton graph convolution-based model that works on a commodity camera mounted onto a moving robot. Our emotion recognition is integrated into a mapless navigation scheme and makes no assumptions about the environment of pedestrian motion. It achieves a mean average emotion prediction precision of 82. 47% on the Emotion-Gait benchmark dataset. We outperform current state-of-art algorithms for emotion recognition from 3D gaits. We highlight its benefits in terms of navigation in indoor scenes using a Clearpath Jackal robot.

ICRA Conference 2020 Conference Paper

RoadTrack: Realtime Tracking of Road Agents in Dense and Heterogeneous Environments

  • Rohan Chandra
  • Uttaran Bhattacharya
  • Tanmay Randhavane
  • Aniket Bera
  • Dinesh Manocha

We present a realtime tracking algorithm, Road-Track, to track heterogeneous road-agents in dense traffic videos. Our approach is designed for dense traffic scenarios that consist of different road-agents such as pedestrians, two-wheelers, cars, buses, etc. sharing the road. We use the tracking-by-detection approach where we track a road-agent by matching the appearance or bounding box region in the current frame with the predicted bounding box region propagated from the previous frame. Roadtrack uses a novel motion model called the Simultaneous Collision Avoidance and Interaction (SimCAI) model to predict the motion of road-agents by modeling collision avoidance and interactions between the road-agents for the next frame. We demonstrate the advantage of RoadTrack on a dataset of dense traffic videos and observe an accuracy of 75. 8% on this dataset, outperforming prior state-of-the-art tracking algorithms by at least 5. 2%. RoadTrack operates in realtime at approximately 30 fps and is at least 4× faster than prior tracking algorithms on standard tracking datasets.

AAAI Conference 2020 Conference Paper

STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits

  • Uttaran Bhattacharya
  • Trisha Mittal
  • Rohan Chandra
  • Tanmay Randhavane
  • Aniket Bera
  • Dinesh Manocha

We present a novel classifier network called STEP, to classify perceived human emotion from gaits, based on a Spatial Temporal Graph Convolutional Network (ST-GCN) architecture. Given an RGB video of an individual walking, our formulation implicitly exploits the gait features to classify the perceived emotion of the human into one of four emotions: happy, sad, angry, or neutral. We train STEP on annotated real-world gait videos, augmented with annotated synthetic gaits generated using a novel generative network called STEP-Gen, built on an ST-GCN based Conditional Variational Autoencoder (CVAE). We incorporate a novel pushpull regularization loss in the CVAE formulation of STEP- Gen to generate realistic gaits and improve the classification accuracy of STEP. We also release a novel dataset (E-Gait), which consists of 4, 227 human gaits annotated with perceived emotions along with thousands of synthetic gaits. In practice, STEP can learn the affective features and exhibits classification accuracy of 88% on E-Gait, which is 14–30% more accurate over prior methods.

IROS Conference 2019 Conference Paper

DensePeds: Pedestrian Tracking in Dense Crowds Using Front-RVO and Sparse Features

  • Rohan Chandra
  • Uttaran Bhattacharya
  • Aniket Bera
  • Dinesh Manocha

We present a pedestrian tracking algorithm, DensePeds, that tracks individuals in highly dense crowds (>2 pedestrians per square meter). Our approach is designed for videos captured from front-facing or elevated cameras. We present a new motion model called Front-RVO (FRVO) for predicting pedestrian movements in dense situations using collision avoidance constraints and combine it with state-of-the-art Mask R-CNN to compute sparse feature vectors that reduce the loss of pedestrian tracks (false negatives). We evaluate DensePeds on the standard MOT benchmarks as well as a new dense crowd dataset. In practice, our approach is 4. 5 × faster than prior tracking algorithms on the MOT benchmark and we are state-of-the-art in dense crowd videos by over 2. 6% on the absolute scale on average.

ICRA Conference 2019 Conference Paper

Pedestrian Dominance Modeling for Socially-Aware Robot Navigation

  • Tanmay Randhavane
  • Aniket Bera
  • Emily Kubin
  • Austin Wang
  • Kurt Gray
  • Dinesh Manocha

We present a Pedestrian Dominance Model (PDM) to identify the dominance characteristics of pedestrians for robot navigation. Through a perception study on a simulated dataset of pedestrians, PDM models the perceived dominance levels of pedestrians with varying motion behaviors corresponding to trajectory, speed, and personal space. At runtime, we use PDM to identify the dominance levels of pedestrians to facilitate socially-aware navigation for the robots. PDM can predict dominance levels from trajectories with ~85% accuracy. Prior studies in psychology literature indicate that when interacting with humans, people are more comfortable around people that exhibit complementary movement behaviors. Our algorithm leverages this by enabling the robots to exhibit complementing responses to pedestrian dominance. We also present an application of PDM for generating dominance-based collision-avoidance behaviors in the navigation of autonomous vehicles among pedestrians. We demonstrate the benefits of our algorithm for robots navigating among tens of pedestrians in simulated environments.

IROS Conference 2018 Conference Paper

Identifying Driver Behaviors Using Trajectory Features for Vehicle Navigation

  • Ernest Cheung
  • Aniket Bera
  • Emily Kubin
  • Kurt Gray
  • Dinesh Manocha

We present a novel approach to automatically identify driver behaviors from vehicle trajectories and use them for safe navigation of autonomous vehicles. We propose a novel set of features that can be easily extracted from car trajectories. We derive a data-driven mapping between these features and six driver behaviors using an elaborate web-based user study. We also compute a summarized score indicating a level of awareness that is needed while driving next to other vehicles. We also incorporate our algorithm into a vehicle navigation simulation system and demonstrate its benefits in terms of safer realtime navigation, while driving next to aggressive or dangerous drivers.

AAAI Conference 2018 Conference Paper

MixedPeds: Pedestrian Detection in Unannotated Videos Using Synthetically Generated Human-Agents for Training

  • Ernest Cheung
  • Anson Wong
  • Aniket Bera
  • Dinesh Manocha

We present a new method for training pedestrian detectors on an unannotated set of images. We produce a mixed reality dataset that is composed of real-world background images and synthetically generated static human-agents. Our approach is general, robust, and makes few assumptions about the unannotated dataset. We automatically extract from the dataset: i) the vanishing point to calibrate the virtual camera, and ii) the pedestrians’ scales to generate a Spawn Probability Map, which is a novel concept that guides our algorithm to place the pedestrians at appropriate locations. After putting synthetic human-agents in the unannotated images, we use these augmented images to train a Pedestrian Detector, with the annotations generated along with the synthetic agents. We conducted our experiments using Faster R-CNN by comparing the detection results on the unannotated dataset performed by the detector trained using our approach and detectors trained with other manually labeled datasets. We showed that our approach improves the average precision by 5-13% over these detectors.

IROS Conference 2018 Conference Paper

The Socially Invisible Robot Navigation in the Social World Using Robot Entitativity

  • Aniket Bera
  • Tanmay Randhavane
  • Emily Kubin
  • Austin Wang
  • Kurt Gray
  • Dinesh Manocha

We present a real-time, data-driven algorithm to enhance the social-invisibility of robots within crowds. Our approach is based on prior psychological research, which reveals that people notice and-importantly-react negatively to groups of social actors when they have high entitativity, moving in a tight group with similar appearances and trajectories. In order to evaluate that behavior, we performed a user study to develop navigational algorithms that minimize entitativity. This study establishes mapping between emotional reactions and multi-robot trajectories and appearances, and further generalizes the finding across various environmental conditions. We demonstrate the applicability of our entitativity modeling for trajectory computation for active surveillance and dynamic intervention in simulated robot-human interaction scenarios. Our approach empirically shows that various levels of entitative robots can be used to both avoid and influence pedestrians while not eliciting strong emotional reactions, giving multi-robot systems socially-invisibility.

IJCAI Conference 2017 Conference Paper

Aggressive, Tense or Shy? Identifying Personality Traits from Crowd Videos

  • Aniket Bera
  • Tanmay Randhavane
  • Dinesh Manocha

We present a real-time algorithm to automatically classify the behavior or personality of a pedestrian based on his or her movements in a crowd video. Our classification criterion is based on Personality Trait theory. We present a statistical scheme that dynamically learns the behavior of every pedestrian and computes its motion model. This model is combined with global crowd characteristics to compute the movement patterns and motion dynamics and use them for crowd prediction. Our learning scheme is general and we highlight its performance in identifying the personality of different pedestrians in low and high density crowd videos. We also evaluate the accuracy by comparing the results with a user study.

IROS Conference 2017 Conference Paper

SocioSense: Robot navigation amongst pedestrians with social and psychological constraints

  • Aniket Bera
  • Tanmay Randhavane
  • Rohan Prinja
  • Dinesh Manocha

We present a real-time algorithm, SocioSense, for socially-aware navigation of a robot amongst pedestrians. Our approach computes time-varying behaviors of each pedestrian using Bayesian learning and Personality Trait theory. These psychological characteristics are used for long-term path prediction and generating proxemic characteristics for each pedestrian. We combine these psychological constraints with social constraints to perform human-aware robot navigation in low- to medium-density crowds. The estimation of time-varying behaviors and pedestrian personalities can improve the performance of long-term path prediction by 21%, as compared to prior interactive path prediction algorithms. We also demonstrate the benefits of our socially-aware navigation in simulated environments with tens of pedestrians.

ICRA Conference 2016 Conference Paper

GLMP- realtime pedestrian path prediction using global and local movement patterns

  • Aniket Bera
  • Sujeong Kim
  • Tanmay Randhavane
  • Srihari Pratapa
  • Dinesh Manocha

We present a novel real-time algorithm to predict the path of pedestrians in cluttered environments. Our approach makes no assumption about pedestrian motion or crowd density, and is useful for short-term as well as long-term prediction. We interactively learn the characteristics of pedestrian motion and movement patterns from 2D trajectories using Bayesian inference. These include local movement patterns corresponding to the current and preferred velocities and global characteristics such as entry points and movement features. Our approach involves no precomputation and we demonstrate the real-time performance of our prediction algorithm on sparse and noisy trajectory data extracted from dense indoor and outdoor crowd videos. The combination of local and global movement patterns can improve the accuracy of long-term prediction by 12–18% over prior methods in high-density videos.

ICRA Conference 2015 Conference Paper

REACH - Realtime crowd tracking using a hybrid motion model

  • Aniket Bera
  • Dinesh Manocha

We present a novel, real-time algorithm to extract the trajectory of each pedestrian in moderately dense crowd videos. In order to improve the tracking accuracy, we use a hybrid motion model that combines discrete and continuous flow models. The discrete model is based on microscopic agent formulation and is used for local navigation, interaction, and collision avoidance. The continuum model accounts for macroscopic behaviors, including crowd orientation and flow. We use our hybrid model with particle filters to compute the trajectories at interactive rates. We demonstrate its performance in moderately-dense crowd videos with tens of pedestrians and highlight the improved accuracy on different datasets.

ICRA Conference 2014 Conference Paper

AdaPT: Real-time adaptive pedestrian tracking for crowded scenes

  • Aniket Bera
  • Nico Galoppo
  • Dillon Sharlet
  • Adam T. Lake
  • Dinesh Manocha

We present a novel realtime algorithm to compute the trajectory of each pedestrian in a crowded scene. Our formulation is based on an adaptive scheme that uses a combination of deterministic and probabilistic trackers to achieve high accuracy and efficiency simultaneously. Furthermore, we integrate it with a multi-agent motion model and local interaction scheme to accurately compute the trajectory of each pedestrian. We highlight the performance and benefits of our algorithm on well-known datasets with tens of pedestrians.

v2026.09.13