Author name cluster

Minh Nhat Vu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

25 papers

2 author rows

IROS Conference 2025 Conference Paper

Action Tokenizer Matters in In-Context Imitation Learning

An Dinh Vuong
Minh Nhat Vu
Dong An 0002
Ian D. Reid 0001

In-context imitation learning (ICIL) is a new paradigm that enables robots to generalize from demonstrations to unseen tasks without retraining. A well-structured action representation is the key to capturing demonstration information effectively, yet action tokenizer (the process of discretizing and encoding actions) remains largely unexplored in ICIL. In this work, we first systematically evaluate existing action tokenizer methods in ICIL and reveal a critical limitation: while they effectively encode action trajectories, they fail to preserve temporal smoothness, which is crucial for stable robotic execution. To address this, we propose LipVQ-VAE, a variational autoencoder that enforces the Lipschitz condition in the latent action space via weight normalization. By propagating smoothness constraints from raw action inputs to a quantized latent codebook, LipVQ-VAE generates smoother actions. When integrating into ICIL, LipVQ-VAE improves performance by more than 5. 3% in high-fidelity simulators, with real-world experiments confirming its ability to produce smoother, more reliable trajectories. Code and checkpoints are available at https://action-tokenizer-matters.github.io/.

IROS Conference 2025 Conference Paper

Efficient Collision Detection for Long and Slender Robotic Links in Euclidean Distance Fields: Application to a Forestry Crane

Marc-Philip Ecker
Bernhard Bischof
Minh Nhat Vu
Christoph Fröhlich
Tobias Glück
Wolfgang Kemmetmüller

Collision-free motion planning in complex outdoor environments relies heavily on perceiving the surroundings through exteroceptive sensors. A widely used approach represents the environment as a voxelized Euclidean distance field, where robots are typically approximated by spheres. However, for large-scale manipulators such as forestry cranes, which feature long and slender links, this conventional spherical approximation becomes inefficient and inaccurate. This work presents a novel collision detection algorithm specifically designed to exploit the elongated structure of such manipulators, significantly enhancing the computational efficiency of motion planning algorithms. Unlike traditional sphere decomposition methods, our approach not only improves computational efficiency but also naturally eliminates the need to fine-tune the approximation accuracy as an additional parameter. We validate the algorithm’s effectiveness using real-world LiDAR data from a forestry crane application, as well as simulated environment data.

JMLR Journal 2025 Journal Article

EMaP: Explainable AI with Manifold-based Perturbations

Minh Nhat Vu
Huy Quang Mai
My T. Thai

In the last few years, many explanation methods based on the perturbations of input data have been introduced to shed light on the predictions generated by black-box models. The goal of this work is to introduce a novel perturbation scheme so that more faithful and robust explanations can be obtained. Our study focuses on the impact of perturbing directions on the data topology. We show that perturbing along the orthogonal directions of the input manifold better preserves the data topology, both in the worst-case analysis of the discrete Gromov-Hausdorff distance and in the average-case analysis via persistent homology. From those results, we introduce EMaP algorithm, realizing the orthogonal perturbation scheme. Our experiments show that EMaP not only improves the explainers' performance but also helps them overcome a recently developed attack against perturbation-based explanation methods. [abs] [ pdf ][ bib ] &copy JMLR 2025. ( edit, beta )

ICRA Conference 2025 Conference Paper

FedEFM: Federated Endovascular Foundation Model with Unseen Data

Tuong Do
Nghia Vu
Tudor Jianu
Baoru Huang
Minh Nhat Vu
Jionglong Su
Erman Tjiputra
Quang D. Tran

In endovascular surgery, the precise identification of catheters and guidewires in X-ray images is essential for reducing intervention risks. However, accurately segmenting catheter and guidewire structures is challenging due to the limited availability of labeled data. Foundation models offer a promising solution by enabling the collection of similar-domain data to train models whose weights can be fine-tuned for downstream tasks. Nonetheless, large-scale data collection for training is constrained by the necessity of maintaining patient privacy. This paper proposes a new method to train a foundation model in a decentralized federated learning setting for endovascular intervention. To ensure the feasibility of the training, we tackle the unseen data issue using differentiable Earth Mover's Distance within a knowledge distillation frame-work. Once trained, our foundation model's weights provide valuable initialization for downstream tasks, thereby enhancing task-specific performance. Intensive experiments show that our approach achieves new state-of-the-art results, contributing to advancements in endovascular intervention and robotic-assisted endovascular surgery, while addressing the critical issue of data sharing in the medical domain.

IROS Conference 2025 Conference Paper

FlowMP: Learning Motion Fields for Robot Planning with Conditional Flow Matching

Khang Nguyen 0003
An T. Le 0001
Tien Pham
Manfred Huber
Jan Peters 0001
Minh Nhat Vu

Prior flow matching methods in robotics have primarily learned velocity fields to morph one distribution of trajectories into another. In this work, we extend flow matching to capture second-order trajectory dynamics, incorporating acceleration effects either explicitly in the model or implicitly through the learning objective. Unlike diffusion models, which rely on a noisy forward process and iterative denoising steps, flow matching trains a continuous transformation (flow) that directly maps a simple prior distribution to the target trajectory distribution without any denoising procedure. By modeling trajectories with second-order dynamics, our approach ensures that the generated robot motions are smooth and physically executable, avoiding the jerky or dynamically infeasible trajectories that first-order models might produce. We empirically demonstrate that this second-order conditional flow matching yields superior performance on motion planning benchmarks, achieving smoother trajectories and higher success rates than baseline planners. These findings highlight the advantage of learning acceleration-aware motion fields, as our method outperforms existing motion planning methods in terms of trajectory quality and planning success. Our source code is available at: https://github.com/mkhangg/flow_mp.

IROS Conference 2025 Conference Paper

Generating Actionable Robot Knowledge Bases by Combining 3D Scene Graphs with Robot Ontologies

Giang Nguyen
Mihai Pomarlan
Sascha Jongebloed
Nils Leusmann
Minh Nhat Vu
Michael Beetz

In robotics, the effective integration of environ-mental data into actionable knowledge remains a significant challenge due to the variety and incompatibility of data formats commonly used in scene descriptions, such as MJCF, URDF, and SDF. This paper presents a novel approach that addresses these challenges by developing a unified scene graph model that standardizes these varied formats into the Universal Scene Description (USD) format. This standardization facilitates the integration of these scene graphs with robot ontologies through semantic reporting, enabling the translation of complex environmental data into actionable knowledge essential for cognitive robotic control. We evaluated our approach by converting procedural 3D environments into USD format, which is then annotated semantically and translated into a knowledge graph to effectively answer competency questions, demonstrating its utility for real-time robotic decision-making. Additionally, we developed a web-based visualization tool to support the semantic mapping process, providing users with an intuitive interface to manage the 3D environment.

IROS Conference 2025 Conference Paper

GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning

Huy Hoang Nguyen
An Vuong
Anh Nguyen 0003
Ian D. Reid 0001
Minh Nhat Vu

Grasp detection is a fundamental robotic task critical to the success of many industrial applications. However, current language-driven models for this task often struggle with cluttered images, lengthy textual descriptions, or slow inference speed. We introduce GraspMamba, a new language-driven grasp detection method that employs hierarchical feature fusion with Mamba vision to tackle these challenges. By leveraging rich visual features of the Mamba-based backbone alongside textual information, our approach effectively enhances the fusion of multimodal features. GraspMamba represents the first Mamba-based grasp detection model to extract vision and language features at multiple scales, delivering robust performance and rapid inference time. Intensive experiments show that GraspMamba outperforms recent methods by a clear margin. We validate our approach through real-world robotic experiments, highlighting its fast inference speed.

IROS Conference 2025 Conference Paper

GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System

Quang Nguyen
Tri Le
Huy Nguyen
Thieu Vo
Tung D. Ta
Baoru Huang
Minh Nhat Vu
Anh Nguyen 0003

Language-driven grasp detection has the potential to revolutionize human-robot interaction by allowing robots to understand and execute grasping tasks based on natural language commands. However, existing approaches face two key challenges. First, they often struggle to interpret complex text instructions or operate ineffectively in densely cluttered environments. Second, most methods require a training or fine-tuning step to adapt to new domains, limiting their generation in real-world applications. In this paper, we introduce GraspMAS, a new multi-agent system framework for language-driven grasp detection. GraspMAS is designed to reason through ambiguities and improve decision-making in real-world scenarios. Our framework consists of three specialized agents: Planner, responsible for strategizing complex queries; Coder, which generates and executes source code; and Observer, which evaluates the outcomes and provides feedback. Intensive experiments on two large-scale datasets demonstrate that our GraspMAS significantly outperforms existing baselines. Additionally, robot experiments conducted in both simulation and real-world settings further validate the effectiveness of our approach. Our project page is available at https://zquang2202.github.io/GraspMAS.

TMLR Journal 2025 Journal Article

Model Tensor Planning

An Thai Le
Khai Nguyen
Minh Nhat Vu
Joao Carvalho
Jan Peters

Sampling-based model predictive control (MPC) offers strong performance in nonlinear and contact-rich robotic tasks, yet often suffers from poor exploration due to locally greedy sampling schemes. We propose \emph{Model Tensor Planning} (MTP), a novel sampling-based MPC framework that introduces high-entropy control trajectory generation through structured tensor sampling. By sampling over randomized multipartite graphs and interpolating control trajectories with B-splines and Akima splines, MTP ensures smooth and globally diverse control candidates. We further propose a simple $\beta$-mixing strategy that blends local exploitative and global exploratory samples within the modified Cross-Entropy Method (CEM) update, balancing control refinement and exploration. Theoretically, we show that MTP achieves asymptotic path coverage and maximum entropy in the control trajectory space in the limit of infinite tensor depth and width. Our implementation is fully vectorized using JAX and compatible with MuJoCo XLA, supporting \emph{Just-in-time} (JIT) compilation and batched rollouts for real-time control with online domain randomization. Through experiments on various challenging robotic tasks, ranging from dexterous in-hand manipulation to humanoid locomotion, we demonstrate that MTP outperforms standard MPC and evolutionary strategy baselines in task success and control robustness. Design and sensitivity ablations confirm the effectiveness of MTP’s tensor sampling structure, spline interpolation choices, and mixing strategy. Altogether, MTP offers a scalable framework for robust exploration in model-based planning and control.

ICRA Conference 2025 Conference Paper

Near Time-Optimal Hybrid Motion Planning for Timber Cranes

Marc-Philip Ecker
Bernhard Bischof
Minh Nhat Vu
Christoph Fröhlich
Tobias Glück
Wolfgang Kemmetmüller

Efficient, collision-free motion planning is essential for automating large-scale manipulators like timber cranes. They come with unique challenges such as hydraulic actuation constraints and passive joints-factors that are seldom addressed by current motion planning methods. This paper introduces a novel approach for time-optimal, collision-free hybrid motion planning for a hydraulically actuated timber crane with passive joints. We enhance the via-point-based stochastic trajectory optimization (VP-STO) algorithm to in-clude pump flow rate constraints and develop a novel collision cost formulation to improve robustness. The effectiveness of the enhanced VP-STO as an optimal single-query global planner is validated by comparison with an informed RRT* algorithm using a time-optimal path parameterization (TOPP). The over-all hybrid motion planning is formed by combination with a gradient-based local planner that is designed to follow the global planner's reference and to systematically consider the passive joint dynamics for both collision avoidance and sway damping.

ICRA Conference 2025 Conference Paper

Online Trajectory Replanner for Dynamically Grasping Irregular Objects

Minh Nhat Vu
Florian Grander
Anh Nguyen 0003
Christoph Unger

This paper presents a new trajectory replanner for grasping irregular objects. Unlike conventional grasping tasks where the object's geometry is assumed simple, we aim to achieve a “dynamic grasp” of the irregular objects, which requires continuous adjustment during the grasping process. To effectively handle irregular objects, we propose a trajectory optimization framework that comprises two phases. Firstly, in a specified time limit of 10 s, initial offline trajectories are computed for a seamless motion from an initial configuration of the robot to grasp the object and deliver it to a predefined target location. Secondly, fast online trajectory optimization is implemented to update robot trajectories in real-time within 100 ms. This helps to mitigate pose estimation errors from the vision system. To account for model inaccuracies, disturbances, and other non-modeled effects, trajectory tracking controllers for both the robot and the gripper are implemented to execute the optimal trajectories from the proposed framework. The intensive experimental results effectively demonstrate the performance of our trajectory planning framework in both simulation and real-world scenarios.

ICRA Conference 2025 Conference Paper

Robotic-CLIP: Fine-Tuning CLIP on Action Data for Robotic Applications

Nghia Nguyen
Minh Nhat Vu
Tung D. Ta
Baoru Huang
Thieu Vo
Ngan Le
Anh Nguyen 0003

Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309, 433 videos (≈ 7. 4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.

ICML Conference 2025 Conference Paper

Topological Signatures of Adversaries in Multimodal Alignments

Minh Nhat Vu
Geigh Zollicoffer
Huy Quang Mai
Ben Nebgen
Boian S. Alexandrov
Manish Bhattarai

Multimodal Machine Learning systems, particularly those aligning text and image data like CLIP/BLIP models, have become increasingly prevalent, yet remain susceptible to adversarial attacks. While substantial research has addressed adversarial robustness in unimodal contexts, defense strategies for multimodal systems are underexplored. This work investigates the topological signatures that arise between image and text embeddings and shows how adversarial attacks disrupt their alignment, introducing distinctive signatures. We specifically leverage persistent homology and introduce two novel Topological-Contrastive losses based on Total Persistence and Multi-scale kernel methods to analyze the topological signatures introduced by adversarial perturbations. We observe a pattern of monotonic changes in the proposed topological losses emerging in a wide range of attacks on image-text alignments, as more adversarial samples are introduced in the data. By designing an algorithm to back-propagate these signatures to input samples, we are able to integrate these signatures into Maximum Mean Discrepancy tests, creating a novel class of tests that leverage topological signatures for better adversarial detection.

ICRA Conference 2025 Conference Paper

Towards Autonomous Wood-Log Grasping with a Forestry Crane: Simulator and Benchmarking

Minh Nhat Vu
Alexander Wachter
Gerald Ebmer
Marc-Philip Ecker
Tobias Glück
Anh Nguyen 0003
Wolfgang Kemmetmüller
Andreas Kugi

Forestry machines operated in forest production environments face challenges when performing manipulation tasks, especially regarding the complicated dynamics of underactuated crane systems and the heavy weight of logs to be grasped. This study investigates the feasibility of using reinforcement learning for forestry crane manipulators in grasping and lifting heavy wood logs autonomously. We first build a simulator using Mujoco physics engine to create realistic scenarios, including modeling a forestry crane with 8 degrees of freedom from CAD data and wood logs of different sizes. We further implement a velocity controller for autonomous log grasping with deep reinforcement learning using a curriculum strategy. Utilizing our new simulator, the proposed control strategy exhibits a success rate of 96% when grasping logs of different diameters and under random initial configurations of the forestry crane. In addition, reward functions and reinforcement learning baselines are implemented to provide an open-source benchmark for the community in large-scale manipulation tasks. A video with several demonstrations can be seen at https://www.acin.tuwien.ac.at/en/d18a/.

ICRA Conference 2024 Conference Paper

Grasp-Anything: Large-scale Grasp Dataset from Foundation Models

Vuong Dinh An
Minh Nhat Vu
Hieu Le
Baoru Huang
Huynh Thi Thanh Binh
Thieu Vo
Andreas Kugi
Anh Nguyen 0003

Foundation models such as ChatGPT have made significant strides in robotic tasks due to their universal representation of real-world domains. In this paper, we leverage foundation models to tackle grasp detection, a persistent challenge in robotics with broad industrial applications. Despite numerous grasp datasets, their object diversity remains limited compared to real-world figures. Fortunately, foundation models possess an extensive repository of real-world knowledge, including objects we encounter in our daily lives. As a consequence, a promising solution to the limited representation in previous grasp datasets is to harness the universal knowledge embedded in these foundation models. We present Grasp-Anything, a new large-scale grasp dataset synthesized from foundation models to implement this solution. Grasp-Anything excels in diversity and magnitude, boasting 1M samples with text descriptions and more than 3M objects, surpassing prior datasets. Empirically, we show that Grasp-Anything successfully facilitates zero-shot grasp detection on vision-based tasks and real-world robotic experiments. Our dataset and code are available at https://airvlab.github.io/grasp-anything/.

IROS Conference 2024 Conference Paper

HabiCrowd: A High Performance Simulator for Crowd-Aware Visual Navigation

An Vuong
Toan Nguyen 0004
Minh Nhat Vu
Baoru Huang
Huynh Thi Thanh Binh
Thieu Vo
Anh Nguyen 0003

Visual navigation, a foundational aspect of Embodied AI (E-AI) and robotics has been extensively studied in the past few years. While many 3D simulators have been introduced for the visual navigation tasks, scarcely works have combined human dynamics, creating the gap between simulation and real-world applications. Furthermore, current 3D simulators incorporating human dynamics have several limitations, particularly in terms of computational efficiency, which is a promise of modern simulators. To overcome these issues, we introduce HabiCrowd, the new standard benchmark for crowd-aware visual navigation that includes a crowd dynamics model with diverse human settings into photorealistic environments. Empirical evaluations demonstrate that our proposed human dynamics model achieves state-of-the-art performance in collision avoidance while exhibiting superior computational efficiency compared to its counterparts. We leverage HabiCrowd to conduct several comprehensive studies on crowd-aware visual navigation tasks and human-robot interactions. The source code and data can be found at https://habicrowd.github.io/.

ICRA Conference 2024 Conference Paper

Language-Conditioned Affordance-Pose Detection in 3D Point Clouds

Toan Nguyen 0004
Minh Nhat Vu
Baoru Huang
Tuan Van Vo
Vy Truong
Ngan Le
Thieu Vo
Bac Le

Affordance detection and pose estimation are of great importance in many robotic applications. Their combination helps the robot gain an enhanced manipulation capability, in which the generated pose can facilitate the corresponding affordance task. Previous methods for affodance-pose joint learning are limited to a predefined set of affordances, thus limiting the adaptability of robots in real-world environments. In this paper, we propose a new method for language-conditioned affordance-pose joint learning in 3D point clouds. Given a 3D point cloud object, our method detects the affordance region and generates appropriate 6-DoF poses for any unconstrained affordance label. Our method consists of an open-vocabulary affordance detection branch and a language-guided diffusion model that generates 6-DoF poses based on the affordance text. We also introduce a new high-quality dataset for the task of language-driven affordance-pose joint learning. Intensive experimental results demonstrate that our proposed method works effectively on a wide range of open-vocabulary affordances and outperforms other baselines by a large margin. In addition, we illustrate the usefulness of our method in real-world robotic applications. Our code and dataset are publicly available at https://3DAPNet.github.io.

IROS Conference 2024 Conference Paper

Language-driven Grasp Detection with Mask-guided Attention

Tuan Van Vo
Minh Nhat Vu
Baoru Huang
An Vuong
Ngan Le
Thieu Vo
Anh Nguyen 0003

Grasp detection is an essential task in robotics with various industrial applications. However, traditional methods often struggle with occlusions and do not utilize language for grasping. Incorporating natural language into grasp detection remains a challenging task and largely unexplored. To address this gap, we propose a new method for language-driven grasp detection with mask-guided attention by utilizing the transformer attention mechanism with semantic segmentation features. Our approach integrates visual data, segmentation mask features, and natural language instructions, significantly improving grasp detection accuracy. Our work introduces a new framework for language-driven grasp detection, paving the way for language-driven robotic applications. Intensive experiments show that our method outperforms other recent baselines by a clear margin, with a 10. 0% success score improvement. We further validate our method in real-world robotic experiments, confirming the effectiveness of our approach.

IROS Conference 2024 Conference Paper

Lightweight Language-driven Grasp Detection using Conditional Consistency Model

Nghia Nguyen
Minh Nhat Vu
Baoru Huang
An Vuong
Ngan Le
Thieu Vo
Anh Nguyen 0003

Language-driven grasp detection is a fundamental yet challenging task in robotics with various industrial applications. This work presents a new approach for language-driven grasp detection that leverages lightweight diffusion models to achieve fast inference time. By integrating diffusion processes with grasping prompts in natural language, our method can effectively encode visual and textual information, enabling more accurate and versatile grasp positioning that aligns well with the text query. To overcome the long inference time problem in diffusion models, we leverage the image and text features as the condition in the consistency model to reduce the number of denoising timesteps during inference. The intensive experimental results show that our method outperforms other recent grasp detection methods and lightweight diffusion models by a clear margin. We further validate our method in real-world robotic experiments to demonstrate its fast inference time capability.

ICRA Conference 2024 Conference Paper

Observer-based Controller Design for Oscillation Damping of a Novel Suspended Underactuated Aerial Platform

Hemjyoti Das
Minh Nhat Vu
Tobias Egle
Christian Ott 0001

In this work, we present a novel actuation strategy for a suspended aerial platform. By utilizing an underactuation approach, we demonstrate the successful oscillation damping of the proposed platform, modeled as a spherical double pendulum. A state estimator is designed in order to obtain the deflection angles of the platform, which uses only onboard IMU measurements. The state estimator is an extended Kalman filter (EKF) with intermittent measurements obtained at different frequencies. An optimal state feedback controller and a PD+ controller are designed in order to dampen the oscillations of the platform in the joint space and task space respectively. The proposed underactuated platform is found to be more energy-efficient than an omnidirectional platform and requires fewer actuators. The effectiveness of our proposed system is validated using both simulations and experimental studies.

ICRA Conference 2024 Conference Paper

Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlation

Tuan Van Vo
Minh Nhat Vu
Baoru Huang
Toan Nguyen 0004
Ngan Le
Thieu Vo
Anh Nguyen 0003

Affordance detection presents intricate challenges and has a wide range of robotic applications. Previous works have faced limitations such as the complexities of 3D object shapes, the wide range of potential affordances on real-world objects, and the lack of open-vocabulary support for affordance understanding. In this paper, we introduce a new open-vocabulary affordance detection method in 3D point clouds, leveraging knowledge distillation and text-point correlation. Our approach employs pre-trained 3D models through knowledge distillation to enhance feature extraction and semantic understanding in 3D point clouds. We further introduce a new text-point correlation method to learn the semantic links between point cloud features and open-vocabulary labels. The intensive experiments show that our approach outperforms previous works and adapts to new affordance labels and unseen objects. Notably, our method achieves the improvement of 7. 96% mIOU score compared to the baselines. Furthermore, it offers real-time inference which is well-suitable for robotic manipulation applications.

IROS Conference 2024 Conference Paper

ProSIP: Probabilistic Surface Interaction Primitives for Learning of Robotic Cleaning of Edges

Christoph Unger
Christian Hartl-Nesic
Minh Nhat Vu
Andreas Kugi

Learning from demonstration (LfD) has emerged as a promising approach enabling robots to acquire complex tasks directly from human demonstrations. However, tasks involving surface interactions on freeform 3D surfaces present unique challenges in modeling and execution, especially when geometric variations exist between demonstrations and robot execution. This paper proposes a novel framework called probabilistic surface interaction primitives (ProSIP), which systematically incorporates the surface path and the local surface features into the learning procedure. An instrumented tool allows seamless recording and execution of human demonstrations. By design, ProSIPs are independent of time, invariant to rigid-body displacements, and apply to any robotic platform with a Cartesian controller. The framework is employed for an edge-cleaning task of bathroom sinks. The generalization capability to various object geometries and significantly distorted objects is demonstrated. Simulations and an experimental setup with a 9-degrees-of-freedom robotic platform confirm the performance.

NeurIPS Conference 2023 Conference Paper

Language-driven Scene Synthesis using Multi-conditional Diffusion Model

An Dinh Vuong
Minh Nhat Vu
Toan Nguyen
Baoru Huang
Dzung Nguyen
Thieu Vo
Anh Nguyen

Scene synthesis is a challenging problem with several industrial applications. Recently, substantial efforts have been directed to synthesize the scene using human motions, room layouts, or spatial graphs as the input. However, few studies have addressed this problem from multiple modalities, especially combining text prompts. In this paper, we propose a language-driven scene synthesis task, which is a new task that integrates text prompts, human motion, and existing objects for scene synthesis. Unlike other single-condition synthesis tasks, our problem involves multiple conditions and requires a strategy for processing and encoding them into a unified space. To address the challenge, we present a multi-conditional diffusion model, which differs from the implicit unification approach of other diffusion literature by explicitly predicting the guiding points for the original data distribution. We demonstrate that our approach is theoretically supportive. The intensive experiment results illustrate that our method outperforms state-of-the-art benchmarks and enables natural scene editing applications. The source code and dataset can be accessed at https: //lang-scene-synth. github. io/.

IROS Conference 2023 Conference Paper

Open-Vocabulary Affordance Detection in 3D Point Clouds

Toan Nguyen 0004
Minh Nhat Vu
An Vuong
Dzung Nguyen
Thieu Vo
Ngan Le
Anh Nguyen 0003

Affordance detection is a challenging problem with a wide variety of robotic applications. Traditional affordance detection methods are limited to a predefined set of affordance labels, hence potentially restricting the adaptability of intelligent robots in complex and dynamic environments. In this paper, we present the Open-Vocabulary Affordance Detection (OpenAD) method, which is capable of detecting an unbounded number of affordances in 3D point clouds. By simultaneously learning the affordance text and the point feature, OpenAD successfully exploits the semantic relationships between affordances. Therefore, our proposed method enables zero-shot detection and can be able to detect previously unseen affordances without a single annotation example. Intensive experimental results show that OpenAD works effectively on a wide range of affordance detection setups and outperforms other baselines by a large margin. Additionally, we demonstrate the practicality of the proposed OpenAD in real-world robotic applications with a fast inference speed. Our project is available at https://openad2023.github.io.

ICRA Conference 2021 Conference Paper

Fast Swing-Up Trajectory Optimization for a Spherical Pendulum on a 7-DoF Collaborative Robot

Minh Nhat Vu
Christian Hartl-Nesic
Andreas Kugi

In this paper, the experimental swing-up of a spherical pendulum mounted on a collaborative robot is presented. The complete mechanical system consists of nine degrees of freedom (DoFs). The primary focus of this work is the design of a fast trajectory planning for the swing-up by systematically incorporating the kinematic and dynamics constraints. The proposed algorithm consists of two steps: First, an offline trajectory optimization is used to build a database of swing-up trajectories, with an average computing time of 10 s for one trajectory. Second, a fast trajectory replanner based on a constrained quadratic program is described, which computes the swing-up trajectory for an arbitrary initial configuration of the system with an average computing time of 0. 2 s. Simulations and experimental results demonstrate the swing-up of the spherical pendulum using a discrete time-variant linear quadratic regulator as a feedback controller.