Author name cluster

Jason Wu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

5 papers

2 author rows

NeurIPS Conference 2025 Conference Paper

ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources

Jason Wu
Yuyang Yuan
Kang Yang
Lance Kaplan
Mani Srivastava

Multimodal deep learning systems are deployed in dynamic scenarios due to the robustness afforded by multiple sensing modalities. Nevertheless, they struggle with varying compute resource availability (due to multi-tenancy, device heterogeneity, etc. ) and fluctuating quality of inputs (from sensor feed corruption, environmental noise, etc. ). Statically provisioned multimodal systems cannot adapt when compute resources change over time, while existing dynamic networks struggle with strict compute budgets. Additionally, both systems often neglect the impact of variations in modality quality. Consequently, modalities suffering substantial corruption may needlessly consume resources better allocated towards other modalities. We propose ADMN, a layer-wise Adaptive Depth Multimodal Network capable of tackling both challenges - it adjusts the total number of active layers across all modalities to meet compute resource constraints, and continually reallocates layers across input modalities according to their modality quality. Our evaluations showcase ADMN can match the accuracy of state-of-the-art networks while reducing up to 75% of their floating-point operations.

PDF Details

IROS Conference 2024 Conference Paper

FlexLoc: Conditional Neural Networks for Zero-Shot Sensor Perspective Invariance in Object Localization with Distributed Multimodal Sensors

Jason Wu
Ziqi Wang 0001
Xiaomin Ouyang
Ho Lyun Jeong
Colin Samplawski
Lance M. Kaplan
Benjamin M. Marlin
Mani Srivastava 0001

Localization is a critical technology for various applications ranging from navigation and surveillance to assisted living. Localization systems typically fuse information from sensors viewing the scene from different perspectives to estimate the target location while also employing multiple modalities for enhanced robustness and accuracy. Recently, such systems have employed end-to-end deep neural models trained on large datasets due to their superior performance and ability to handle data from diverse sensor modalities. However, such neural models are often trained on data collected from a particular set of sensor poses (i. e. , locations and orientations). During real-world deployments, slight deviations from these sensor poses can result in extreme inaccuracies. To address this challenge, we introduce FlexLoc, which employs conditional neural networks to inject node perspective information to adapt the localization pipeline. Specifically, a small subset of model weights are derived from node poses at run time, enabling accurate generalization to unseen perspectives with minimal additional overhead. Our evaluations on a multimodal, multi-view indoor tracking dataset showcase that FlexLoc improves the localization accuracy by almost 50% in the zero-shot case (no calibration data available) compared to the baselines. The source code of FlexLoc is available in https://github.com/nesl/FlexLoc.

Details

ICRA Conference 2023 Conference Paper

Neural-Kalman GNSS/INS Navigation for Precision Agriculture

Yayun Du
Swapnil Sayan Saha
Sandeep Singh Sandha
Arthur Lovekin
Jason Wu
S. Siddharth
Mahesh Chowdhary
Mohammad Khalid Jawed

Precision agricultural robots require high-resolution navigation solutions. In this paper, we introduce a robust neural-inertial sequence learning approach to track such robots with ultra-intermittent GNSS updates. First, we propose an ultra-lightweight neural-Kalman filter that can track agricultural robots within 1. 4 m (1. 4–5. 8× better than competing techniques), while tracking within 2. 75 m with 20 mins of GPS outage. Second, we introduce a user-friendly video-processing toolbox to generate high-resolution (±5 cm) position data for fine-tuning pre-trained neural-inertial models in the field. Third, we introduce the first and largest (6. 5 hours, 4. 5 km, 3 phases) public neural-inertial navigation dataset for precision agricultural robots. The dataset, toolbox, and code are available at: https://github.com/nesl/agrobot.

Details

NeurIPS Conference 2021 Conference Paper

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Paul Pu Liang
Yiwei Lyu
Xiang Fan
Zetian Wu
Yun Cheng
Jason Wu
Leslie (Yufan) Chen
Peter Wu

Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark for multimodal learning spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning spanning innovations in fusion paradigms, optimization objectives, and training approaches. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal machine learning research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized implementations, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.

PDF Details

AAMAS Conference 2018 Conference Paper

NADiA - Towards Neural Network Driven Virtual Human Conversation Agents

Jason Wu
Sayan Ghosh
Mathieu Chollet
Steven Ly
Sharon Mozgai
Stefan Scherer

Advances in artificial intelligence and machine learning - in particular neural networks - have given rise to a new generation of virtual assistants and chatbots. Within this work, we describe the motivation and architecture of NADiA - Neurally Animated Dialog Agent which leverages both the user’s verbal input and facial expressions for multi-modal conversation. NADiA combines a neural language model that generates conversational responses, a convolutional neural network for facial expression analysis, and virtual human technology that is deployed on a mobile phone.

PDF