Author name cluster

Ran Tian

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

8 papers

2 author rows

TMLR Journal 2025 Journal Article

Wolf: Dense Video Captioning with a World Summarization Framework

Boyi Li
Ligeng Zhu
Ran Tian
Shuhan Tan
Yuxiao Chen
Yao Lu
Yin Cui
Sushant Veer

We propose Wolf, a WOrLd summarization Framework for accurate video captioning. Wolf is an automated captioning framework that adopts a mixture-of-experts approach, leveraging complementary strengths of Vision Language Models (VLMs). By utilizing both image and video models, our framework captures different levels of information and summarizes them efficiently. Our approach can be applied to enhance video understanding, auto-labeling, and captioning. To evaluate caption quality, we introduce CapScore, an LLM-based metric to assess the similarity and quality of generated captions compared to the ground truth captions. We further build four human-annotated datasets in three domains: autonomous driving, general scenes, and robotics, to facilitate comprehensive comparisons. We show that Wolf achieves superior captioning performance compared to state-of-the-art approaches from the research community (VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For instance, in comparison with GPT-4V, Wolf improves CapScore (caption quality) by 55.6% and CapScore (caption similarity) by 77.4% on challenging driving videos. Finally, we establish a benchmark for video captioning and introduce a leaderboard, aiming to accelerate advancements in video understanding, captioning, and data alignment.

PDF Details

ICRA Conference 2024 Conference Paper

Open X-Embodiment: Robotic Learning Datasets and RT-X Models: Open X-Embodiment Collaboration

Abby O'Neill
Abdul Rehman
Abhiram Maddukuri
Abhishek Gupta 0004
Abhishek Padalkar
Abraham Lee
Acorn Pooley
Agrim Gupta

Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x. github.io.

Details

AAMAS Conference 2024 Conference Paper

Quantifying Agent Interaction in Multi-agent Reinforcement Learning for Cost-efficient Generalization

Yuxin Chen
Chen Tang
Ran Tian
Chenran Li
Jinning Li
Masayoshi Tomizuka
Wei Zhan

Generalization in Multi-agent Reinforcement Learning (MARL) is challenging. Introducing a diverse set of co-play agents typically boosts the agent’s generalization to unseen co-players. However, the extent to which an agent is influenced by co-players varies across scenarios and environments; thus, the improvement in generalization introduced by diversifying co-players also varies. In this work, we introduce Level of Influence (LoI), a novel metric measuring the interaction intensity among agents within a given scenario and environment. We show that LoI can effectively predict the disparities in the benefits of diversifying co-player distribution across scenarios, offering insights into optimizing training cost for varied situations. The code is available at: https: //github. com/ ThomasChen98/Level-of-Influence.

PDF

ICRA Conference 2022 Conference Paper

Cost-Effective Sensing for Goal Inference: A Model Predictive Approach

Ran Tian
Nan Li 0015
Anouck Girard
Ilya V. Kolmanovsky
Masayoshi Tomizuka

Goal inference is of great importance for a variety of applications that involve interaction, coordination, and/or competition with goal-oriented agents. Typical goal inference approaches use as many pointwise measurements of the agent's trajectory as possible to pursue a most accurate a-posteriori estimate of the goal. However, taking frequent measurements may not be preferred in situations where sensing is associated with high cost (e. g. , sensing + perception may involve high computational/bandwidth cost and sensing may raise security concerns in privacy-critical/data-sensitive applications). In such situations, a sensible tradeoff between the information gained from measurements and the cost associated with sensing actions is highly desirable. This paper introduces a cost-effective sensing strategy for goal inference tasks based on hybrid Kalman filtering and model predictive control. Our key insights include: 1) a model predictive approach can be used to predict the amount of information gained from new measurements over a horizon and thus to optimize the tradeoff between information gain and sensing action cost, and 2) the high computational efficiency of hybrid Kalman filtering can ensure real-time feasibility of such a model predictive approach. We evaluate the proposed cost-effective sensing approach in a goal-oriented task, where we show that compared to standard goal inference approaches, our approach takes a considerably reduced number of measurements while not impairing the speed, accuracy, and reliability of goal inference by taking measurements smartly.

Details

ICRA Conference 2022 Conference Paper

Safety Assurances for Human-Robot Interaction via Confidence-aware Game-theoretic Human Models

Ran Tian
Liting Sun
Andrea Bajcsy
Masayoshi Tomizuka
Anca D. Dragan

An outstanding challenge with safety methods for human-robot interaction is reducing their conservatism while maintaining robustness to variations in human behavior. In this work, we propose that robots use confidence-aware game-theoretic models of human behavior when assessing the safety of a human-robot interaction. By treating the influence between the human and robot as well as the human's rationality as unobserved latent states, we succinctly infer the degree to which a human is following the game-theoretic interaction model. We leverage this model to restrict the set of feasible human controls during safety verification, enabling the robot to confidently modulate the conservatism of its safety monitor online. Evaluations in simulated human-robot scenarios and ablation studies demonstrate that imbuing safety monitors with confidence-aware game-theoretic models enables both safe and efficient human-robot interaction. Moreover, evaluations with real traffic data show that our safety monitor is less conservative than traditional safety methods in real human driving scenarios.

Details

ICRA Conference 2021 Conference Paper

Anytime Game-Theoretic Planning with Active Reasoning About Humans' Latent States for Human-Centered Robots

Ran Tian
Liting Sun
Masayoshi Tomizuka
David Isele

A human-centered robot needs to reason about the cognitive limitation and potential irrationality of its human partner to achieve seamless interactions. This paper proposes an anytime game-theoretic planner that integrates iterative reasoning models, a partially observable Markov decision process, and chance-constrained Monte-Carlo belief tree search for robot behavioral planning. Our planner enables a robot to safely and actively reason about its human partner’s latent cognitive states (bounded intelligence and irrationality) in real-time to maximize its utility better. We validate our approach in an autonomous driving domain where our behavioral planner and a low-level motion controller hierarchically control an autonomous car to negotiate traffic merges. Simulations and user studies are conducted to show our planner’s effectiveness.

Details

AAAI Conference 2021 Conference Paper

Bounded Risk-Sensitive Markov Games: Forward Policy Design and Inverse Reward Learning with Iterative Reasoning and Cumulative Prospect Theory

Ran Tian
Liting Sun
Masayoshi Tomizuka

Classical game-theoretic approaches for multi-agent systems in both the forward policy design problem and the inverse reward learning problem often make strong rationality assumptions: agents perfectly maximize expected utilities under uncertainties. Such assumptions, however, substantially mismatch with observed human behaviors such as satisficing with sub-optimal, risk-seeking, and loss-aversion decisions. Drawing on iterative reasoning models and cumulative prospect theory, we propose a new game-theoretic framework, bounded risk-sensitive Markov Game (BRSMG), that captures two aspects of realistic human behaviors: bounded intelligence and risk-sensitivity. General solutions to both the forward policy design problem and the inverse reward learning problem are provided with theoretical analysis and simulation verification. We validate the proposed forward policy design algorithm and the inverse reward learning algorithm in a two-player navigation scenario. The results show that agents demonstrate bounded-intelligence, risk-averse and risk-seeking behaviors in our framework. Moreover, in the inverse reward learning task, the proposed bounded risk-sensitive inverse learning algorithm outperforms a baseline risk-neutral inverse learning algorithm by effectively learning not only more accurate reward values but also the intelligence levels and the risk-measure parameters of agents from demonstrations.

PDF Details

IROS Conference 2021 Conference Paper

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

Ran Tian
Masayoshi Tomizuka
Liting Sun

Reward function, as an incentive representation that recognizes humans’ agency and rationalizes humans’ actions, is particularly appealing for modeling human behavior in human-robot interaction. Inverse Reinforcement Learning is an effective way to retrieve reward functions from demonstrations. However, it has always been challenging when applying it to multi-agent settings since the mutual influence between agents has to be appropriately modeled. To tackle this challenge, previous work either exploits equilibrium solution concepts by assuming humans as perfectly rational optimizers with unbounded intelligence or pre-assigns humans’ interaction strategies a priori. In this work, we advocate that humans are bounded rational and have different intelligence levels when reasoning about others’ decision-making process, and such an inherent and latent characteristic should be accounted for in reward learning algorithms. Hence, we exploit such insights from Theory-of-Mind and propose a new multi-agent Inverse Reinforcement Learning framework that reasons about humans’ latent intelligence levels during learning. We validate our approach in both zero-sum and general-sum games with synthetic agents, and illustrate a practical application to learning human drivers’ reward functions from real driving data. We compare our approach with two baseline algorithms. The results show that by reasoning about humans’ latent intelligence levels, the proposed approach has more flexibility and capability to retrieve reward functions that explain humans’ driving behaviors better.

Details