Tao Yu

JBHI Journal 2026 Journal Article

APSevLM: Acute Pancreatitis Severity Language Model

• Leqi Zheng
• Jiajun Fang
• Hongyi Chen
• Naiqing Li
• Yunyuan Huang
• Qiulin Ge
• Yang Gu
• Tao Yu

Approximately one-fifth of patients with acute pancreatitis (AP) develop severe forms, which are associated with high mortality rates, making early prediction of severity crucial for effective patient management. In this study, we present APSevLM (Acute Pancreatitis Severity Language Model), a large language model (LLM)-based approach that integrates admission-time clinical data, imaging reports, and expert knowledge to predict AP severity at an early stage. Through a comprehensive evaluation using data from over five hundred patients, APSevLM outperforms traditional scoring systems (BISAP and MCTSI), conventional machine learning algorithms, and state-of-the-art deep learning models, achieving an AUC of 0. 857. Attention visualizations of the model explain complex mechanisms that dynamically weigh different information modalities based on case severity. Furthermore, a systematic feature importance analysis identifies key predictive factors, particularly hematological parameters and cardiac markers, offering valuable insights for clinical practice. Our study positions APSevLM as an accurate predictive model and highlights potential biomarkers for the early diagnosis of severe AP.

TMLR Journal 2026 Journal Article

BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions

• Tao Yu
• Zhengbo Zhang
• Zhiheng Lyu
• Junhao Gong
• Hongzhu Yi
• Xinming Wang
• Yuxuan Zhou
• Jiabing Yang

Efficiently solving real-world problems with LLMs increasingly hinges on their ability to interact with dynamic web environments and autonomously acquire external information. While recent research like Search-R1 and WebDancer demonstrates strong performance in solving web tasks, they heavily rely on additional tools to convert the interactive web environment into static text content. This is in contrast to human browsing behaviors, which involve diverse interactions with the browser, such as scrolling, clicking, and typing. In this paper, we propose BrowserAgent, a more interactive agent that solves complex tasks through human-inspired browser actions. BrowserAgent operates directly on raw web pages via Playwright through a set of predefined browser actions. We adopt a two-stage training (Supervised Fine-Tuning (SFT) and Rejection Fine-Tuning (RFT)) to improve the model's generalization abilities. Despite using significantly less training data than Search-R1, BrowserAgent achieves more competitive results across different Open-QA tasks. Additionally, we introduce an explicit memory mechanism to store key conclusions across steps, further enhancing the model's reasoning capabilities for long-horizon tasks. Notably, BrowserAgent-7B can achieve around 20\% improvement over Search-R1 on multi-hop QA tasks like HotpotQA, 2Wiki, and Bamboogle. These results indicate that BrowserAgent can serve as a more advanced framework for more interactive and scalable web agents.

TMLR Journal 2026 Journal Article

Diffusion Models for Solving Inverse Problems via Posterior Sampling with Piecewise Guidance

• Saeed Mohseni-Sehdeh
• Walid Saad
• Kei Sakaguchi
• Tao Yu

Diffusion models are powerful tools for sampling from high-dimensional distributions by progressively transforming pure noise into structured data through a denoising process. When equipped with a guidance mechanism, these models can also generate samples from conditional distributions. In this paper, a novel diffusion-based framework is introduced for solving inverse problems using a piecewise guidance scheme. The guidance term is defined as a piecewise function of the diffusion timestep, facilitating the use of different approximations during high-noise and low-noise phases. This design is shown to effectively balance computational efficiency with the accuracy of the guidance term. Unlike task-specific approaches that require retraining for each problem, the proposed method is problem-agnostic and readily adaptable to a variety of inverse problems. Additionally, it explicitly incorporates measurement noise into the reconstruction process. The effectiveness of the proposed framework is demonstrated through extensive experiments on image restoration tasks, specifically image inpainting and super-resolution. Using a class conditional diffusion model for recovery, compared to the pseudoinverse-guided diffusion model ($\Pi$GDM) baseline, the proposed framework achieves a reduction in inference time of $25\%$ for inpainting with both random and center masks, and $23\%$ and $24\%$ for $4\times$ and $8\times$ super-resolution tasks, respectively, while incurring only negligible loss in PSNR and SSIM.

JBHI Journal 2026 Journal Article

Dual-Cross Tri-Level Routing Transformer Based Metric Learning Network for Epileptic Seizure Prediction Using a Single-Channel iEEG

• Yifan Wang
• Weidong Yan
• Yulan Ma
• Liang Qiao
• Tao Yu
• Jingyu Liu

With the development of deep brain stimulation technique, single-channel intracranial electroencephalography (iEEG) based seizure prediction is a necessary and urgent needed tool for epilepsy closed-loop neuromodulation. However, previous prediction methods based on multi-channel scalp signals heavily relied on the spatial information, failing to fully exploit the interdependencies between temporal scales and spectral rhythms of single-channel iEEG. Additionally, current contrastive learning strategies can lead to model overfitting by excessively learning the feature distances in small samples, limiting the precision of seizure prediction. To tackle above issues, based on a single-channel iEEG, we propose a novel dual-cross tri-level routing transformer based metric learning network (DC-TRT-MLNet) for epileptic seizure prediction. First, a scale-rhythm dual-cross (DC) graph attention network is introduced to construct the dependent relationships across multi-scale temporal and multi-rhythm spectral features. Second, we design a tri-level routing transformer (TRT) network to comprehensively refine the most seizure-potential routing features while eliminating redundant information. Finally, a hard triplet optimization based metric learning (ML) strategy is developed to iteratively optimize the intra-class and inter-class distances of inter-ictal and pre-ictal routing features. Competitive experimental results on a private Xuanwu Single-Channel iEEG dataset validate the effectiveness of our proposed method, demonstrating the superior prediction performance of our DC-TRT-MLNet compared with the state-of-the-art methods. Our study may offer a new solution for intracranial single-channel seizure prediction.

AAAI Conference 2026 Conference Paper

Dynamic Deep Graph Learning for Incomplete Multi-View Clustering with Masked Graph Reconstruction Loss

• Zhenghao Zhang
• Jun Xie
• Xingchen Chen
• Tao Yu
• Hongzhu Yi
• Kaixin Xu
• Yuanxiang Wang
• Tianyu Zong

The prevalence of real-world multi-view data makes incomplete multi-view clustering (IMVC) a crucial research. The rapid development of Graph Neural Networks (GNNs) has established them as one of the mainstream approaches for multi-view clustering. Despite significant progress in GNNs-based IMVC, some challenges remain: (1) Most methods rely on the K-Nearest Neighbors (KNN) algorithm to construct static graphs from raw data, which introduces noise and diminishes the robustness of the graph topology. (2) Existing methods typically utilize the Mean Squared Error (MSE) loss between the reconstructed graph and the sparse adjacency graph directly as the graph reconstruction loss, leading to substantial gradient noise during optimization. To address these issues, we propose a novel Dynamic Deep Graph Learning for Incomplete Multi-View Clustering with Masked Graph Reconstruction Loss (DGIMVCM). Firstly, we construct a missing-robust global graph from the raw data. A graph convolutional embedding layer is then designed to extract primary features and refined dynamic view-specific graph structures, leveraging the global graph for imputation of missing views. This process is complemented by graph structure contrastive learning, which identifies consistency among view-specific graph structures. Secondly, a graph self-attention encoder is introduced to extract high-level representations based on the imputed primary features and view-specific graphs, and is optimized with a masked graph reconstruction loss to mitigate gradient noise during optimization. Finally, a clustering module is constructed and optimized through a pseudo-label self-supervised training mechanism. Extensive experiments on multiple datasets validate the effectiveness and superiority of DGIMVCM.

AAAI Conference 2026 Conference Paper

Improving Deepfake Detection with Reinforcement Learning-Based Adaptive Data Augmentation

• Yuxuan Chou
• Tao Yu
• Wen Huang
• ZhangYuHeng
• Tao Dai
• Shu-Tao Xia

The generalization capability of deepfake detectors is crucial for real-world applications. Data augmentation to generate synthetic fake faces has served as an effective strategy to enhance generalization. Interestingly, current state-of-the-art (SoTA) methods rely on fixed augmentation strategies, raising a fundamental question: Can a single static augmentation approach suffice, or does the diversity of forgery features necessitate dynamic strategies? We argue that existing methods overlook the evolving complexity of real-world forgery patterns, such as facial warping, expression manipulation, and compression artifacts, which cannot be fully simulated by fixed policies. To bridge this gap, we propose CRDA (Curriculum Reinforcement-Learning Data Augmentation), a novel framework that guides the detector to progressively master multi-domain forgery features from simple to complex. CRDA synthesizes augmented samples using a configurable pool of forgery operations and dynamically generates adversarial samples tailored to the detector’s current learning state. Key to our approach is the integration of reinforcement learning (RL) and causal inference. To efficiently explore the vast augmentation space, an RL agent dynamically selects augmentation actions based on the detector’s performance, ensuring continuous adaptation to increasingly challenging forgeries. Simultaneously, the agent’s output is designed to introduce variations in action spaces, generating heterogeneous forgery patterns. These variations are guided by causal inference theory, which mitigates spurious correlations by suppressing task-irrelevant biases and enforcing the model to focus on causally invariant features. This integration ensures robust generalization by decoupling synthetic augmentation patterns from the model’s learned representations. Extensive experiments demonstrate that the proposed method significantly improves the generalizability of the detector, achieving superior performance compared to state-of-the-art methods on multiple cross-domain datasets.

AAAI Conference 2026 Conference Paper

Improving Generalization in LLM Structured Pruning via Function-Aware Neuron Grouping

• Tao Yu
• Yongqi An
• Kuan Zhu
• Guibo Zhu
• Ming Tang
• Jinqiao Wang

Large Language Models (LLMs) demonstrate impressive performance across natural language tasks but incur substantial computational and storage costs due to their scale. Post-training structured pruning offers an efficient solution. However, when few-shot calibration sets fail to adequately reflect the pretraining data distribution, existing methods exhibit limited generalization to downstream tasks. To address this issue, we propose Function-Aware Neuron Grouping (FANG), a post-training pruning framework that alleviates calibration bias by identifying and preserving neurons critical to specific function. FANG groups neurons with similar function based on the type of semantic context they process and prunes each group independently. During importance estimation within each group, tokens that strongly correlate with the functional role of the neuron group are given higher weighting. Additionally, FANG also preserves neurons that contribute across multiple context types. To achieve a better trade-off between sparsity and performance, it allocates sparsity to each block adaptively based on its functional complexity. Experiments show that FANG improves downstream accuracy while preserving language modeling performance. It achieves the state-of-the-art (SOTA) results when combined with FLAP and OBC, two representative pruning methods. Specifically, FANG outperforms FLAP and OBC by 1.5%–8.5% in average accuracy under 30% and 40% sparsity.

AAAI Conference 2026 Conference Paper

Monocular Mesh Recovery and Body Measurement of Female Saanen Goats

• Bo Jin
• ShichaoZhao
• Jin Lyu
• Bin Zhang
• Tao Yu
• Liang An
• Yebin Liu
• Meili Wang

The lactation performance of Saanen dairy goats, renowned for their high milk yield, is intrinsically linked to their body size, making accurate 3D body measurement essential for assessing milk production potential, yet existing reconstruction methods lack goat-specific authentic 3D data. To address this limitation, we establish the FemaleSaanenGoat dataset containing synchronized eight-view RGBD videos of 55 female Saanen goats (6-18 months). Using multi-view DynamicFusion, we fuse noisy, non-rigid point cloud sequences into high-fidelity 3D scans, overcoming challenges from irregular surfaces and rapid movement. Based on these scans, we develop SaanenGoat, a parametric 3D shape model specifically designed for female Saanen goats. This model features a refined template with 41 skeletal joints and enhanced udder representation, registered with our scan data. A comprehensive shape space constructed from 48 goats enables precise representation of diverse individual variations. With the help of SaanenGoat model, we get high-precision 3D reconstruction from single-view RGBD input, and achieve automated measurement of six critical body dimensions: body length, height, chest width, chest girth, hip width, and hip height. Experimental results demonstrate the superior accuracy of our method in both 3D reconstruction and body measurement, presenting a novel paradigm for large-scale 3D vision applications in precision livestock farming.

ICML Conference 2025 Conference Paper

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

• Yifan Zhang 0004
• Tao Yu
• Haochen Tian 0001
• Chaoyou Fu
• Peiyan Li 0001
• Jianshu Zeng
• Wulin Xie
• Yang Shi 0009

Existing efforts to align multimodal large language models (MLLMs) with human preferences have only achieved progress in narrow areas, such as hallucination reduction, but remain limited in practical applicability and generalizability. To this end, we introduce MM-RLHF, a dataset containing 120k fine-grained, human-annotated preference comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation granularity, and quality. Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce the Critique-Based Reward Model, which generates critiques of model outputs before assigning scores, offering enhanced interpretability and more informative feedback compared to traditional scalar reward mechanisms. Additionally, we propose Dynamic Reward Scaling, a method that adjusts the loss weight of each sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs. Our approach is rigorously evaluated across 10 distinct dimensions, encompassing 27 benchmarks, with results demonstrating significant and consistent improvements in model performance (Figure. 1).

NeurIPS Conference 2025 Conference Paper

OpenCUA: Open Foundations for Computer-Use Agents

• Xinyuan Wang
• Bowen Wang
• Dunjie Lu
• Junlin Yang
• Tianbao Xie
• Junli Wang
• Jiaqi Deng
• Xiaole Guo

Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed. As these agents will increasingly mediate digital interactions and execute consequential decisions on our behalf, the research community needs access to open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose OpenCUA, a comprehensive open-source framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AgentNet, the first large-scale computer-use task dataset spanning 3 operating systems and 200+ applications and websites; (3) a scalable pipeline that transforms demonstrations into state–action pairs with reflective long Chain-of-Thought reasoning that sustain robust performance gains as data scales. Our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OpenCUA-72B achieves an average success rate of 45. 0% on OSWorld‑Verified, establishing a new state-of-the-art (SOTA) among open-source models. Further analysis confirms that our approach generalizes well across domains and benefits significantly from increased test-time computation. We release our annotation tool, datasets, code, and models to build open foundations for further CUA research.

NeurIPS Conference 2025 Conference Paper

Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis

• Tianbao Xie
• Jiaqi Deng
• Xiaochuan Li
• Junlin Yang
• Haoyuan Wu
• Jixuan Chen
• Wenjing Hu
• Xinyuan Wang

Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks with state-of-the-art performance, improving from 23% to 51% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https: //osworld-grounding. github. io.

YNIMG Journal 2025 Journal Article

The changes in neural complexity and connectivity in thalamocortical and cortico-cortical systems after propofol-induced unconsciousness in different temporal scales

• Zhenhu Liang
• Luxin Fan
• Bin Zhang
• Wei Shu
• Duan Li
• Xiaoli Li
• Tao Yu

Existing studies have indicated neural activity across diverse temporal and spatial scales. However, the alterations in complexity, functional connectivity, and directional connectivity within the thalamocortical and corticocortical systems across various scales during propofol-induced unconsciousness remain uncertain. We analyzed the stereo-electroencephalography (SEEG) from wakefulness to unconsciousness among the brain regions of the prefrontal cortex, temporal lobe, and anterior nucleus of the thalamus. The complexity (examined by permutation entropy (PE)), functional connectivity (permutation mutual information (PMI)), and directional connectivity (symbolic conditional mutual information (SCMI) and directionality index (DI)) were calculated across various scales. In the lower-band frequency (0.1-45 Hz) SEEG, after the loss of consciousness, PE significantly decreased (p < 0.001) in all regions and scales, except for the thalamus, which remained relatively unchanged at large scales (τ=32 ms). Following the loss of consciousness, inter-regional PMI either significantly increased or remained stable across different scales (τ=4 ms to 32 ms). During the unconscious state, SCMI between brain regions exhibited inconsistent changes across scales. In the late unconscious stage, the inter-regional DI across all scales indicated a shift from a balanced state of information flow between brain regions to a pattern where the prefrontal cortex and thalamus drive the temporal lobe. Our findings demonstrate that propofol-induced unconsciousness is associated with reduced cortical complexity, diverse functional connectivity, and a disrupted balance of information integration among thalamocortical and cortico-cortical systems. This study enhances the theoretical understanding of anesthetic-induced loss of consciousness by elucidating the scale- and region-specific effects of propofol on thalamocortical and cortico-cortical systems.

AAAI Conference 2025 Conference Paper

View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection

• Qi Zhang
• Zhouhang Luo
• Tao Yu
• Hui Huang

View transformation robustness (VTR) is critical for deep-learning-based multi-view 3D object reconstruction models, which indicates the methods' stability under inputs with various view transformations. However, existing research seldom focused on view transformation robustness in multi-view 3D object reconstruction. One direct way to improve the models' VTR is to produce data with more view transformations and add them to model training. Recent progress on large vision models, particularly Stable Diffusion models, has provided great potential for generating 3D models or synthesizing novel view images with only a single image input. Directly deploying these models at inference consumes heavy computation resources and their robustness to view transformations is not guaranteed either. To fully utilize the power of Stable Diffusion models without extra inference computation burdens, we propose to generate novel views with Stable Diffusion models for better view transformation robustness. Instead of synthesizing random views, we propose a reconstruction error-guided view selection method, which considers the reconstruction errors' spatial distribution of the 3D predictions and chooses the views that could cover the reconstruction errors as much as possible. The methods are trained and tested on sets with large view transformations to validate the 3D reconstruction models' robustness to view transformations. Extensive experiments demonstrate that the proposed method can outperform state-of-the-art 3D reconstruction methods and other view transformation robustness comparison methods.

EAAI Journal 2024 Journal Article

A semi-supervised load identification method with class incremental learning

• Leixin Qiu
• Tao Yu
• Chaofan Lan

With the proposal of the carbon peaking and neutrality targets, non-intrusive load monitoring (NILM) is crucial for energy saving and demand response. How to achieve accurate load identification (LI) is the key to the application of NILM. However, it faces a major challenge: how to identify loads accurately from massive incremental unlabeled data streams. To tackle this challenge, we propose a novel method that combines class incremental learning (CIL) and semi-supervised learning (SSL). Our method prevents catastrophic forgetting by preserving samples, distilling knowledge and aligning weights in incremental tasks. Moreover, our method leverages a semi-supervised learning structure called the Temporal Ensembling to exploit unlabeled data and overcome the semi-supervised problem in incremental learning. We test our method on PLAID and WHITED public datasets and demonstrate its effectiveness.

ICML Conference 2024 Conference Paper

Collage: Light-Weight Low-Precision Strategy for LLM Training

• Tao Yu
• Gaurav Gupta
• Karthick Gopalswamy
• Amith R. Mamidala
• Hao Zhou 0036
• Jeffrey Huynh
• Youngsuk Park
• Ron Diamant

Large models training is plagued by the intense compute cost and limited hardware memory. A practical solution is low-precision representation but is troubled by loss in numerical accuracy and unstable training rendering the model less useful. We argue that low-precision floating points can perform well provided the error is properly compensated at the critical locations in the training process. We propose Collage which utilizes multi-component float representation in low-precision to accurately perform operations with numerical errors accounted. To understand the impact of imprecision to training, we propose a simple and novel metric which tracks the lost information during training as well as differentiates various precision strategies. Our method works with commonly used low-precision such as half-precision ($16$-bit floating points) and can be naturally extended to work with even lower precision such as $8$-bit. Experimental results show that pre-training using Collage removes the requirement of using $32$-bit floating-point copies of the model and attains similar/better training performance compared to $(16, 32)$-bit mixed-precision strategy, with up to $3. 7\times$ speedup and $\sim 15%$ to $23%$ less memory usage in practice. The code is available at https: //github. com/amazon-science/collage.

IROS Conference 2024 Conference Paper

Dynamic SpectraFormer for Ultra-High-Definition Underwater Image Enhancement

• Zhiqiang Hu
• Tao Yu
• Shouren Huang
• Masatoshi Ishikawa

Underwater images suffer from color distortion, haze, and poor visibility due to light refraction and absorption in water. These challenges significantly impact the utilization of Autonomous Underwater Vehicles (AUVs) or marine robots. Typically, color and brightness distortions manifest at lower frequencies, while edge and texture distortions are prevalent at higher frequencies. Traditional methods struggle to concurrently rectify these mixed distortions as they primarily concentrate on the spatial domain. To address these issues, we introduce the Dynamic SpectraFormer, which enhances under-water images through a frequency domain transformer. The Dynamic SpectraFormer introduces an ultra-high-resolution sparse spectrum attention module, which could capture the long-term dependency without losing the universal approximating power. Additionally, we have developed a dynamic spectrum weight generation layer that serves as an adaptive spectrum band selector, accentuating critical frequency bands and suppressing less relevant ones. Consequently, this method significantly improves underwater image quality by addressing both high-and low-frequency distortions. Our extensive ablation studies and comparative evaluations consolidate the Dynamic SpectraFormer’s efficacy across multiple underwater image enhancement benchmarks. The source code is available at https://github.com/arifence2024/DynamicSpectraFormer.git.

AAAI Conference 2024 Conference Paper

Fluctuation-Based Adaptive Structured Pruning for Large Language Models

• Yongqi An
• Xu Zhao
• Tao Yu
• Ming Tang
• Jinqiao Wang

Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.

AAAI Conference 2024 Conference Paper

Neural Physical Simulation with Multi-Resolution Hash Grid Encoding

• Haoxiang Wang
• Tao Yu
• Tianwei Yang
• Hui Qiao
• Qionghai Dai

We explore the generalization of the implicit representation in the physical simulation task. Traditional time-dependent partial differential equations (PDEs) solvers for physical simulation often adopt the grid or mesh for spatial discretization, which is memory-consuming for high resolution and lack of adaptivity. Many implicit representations like local extreme machine or Siren are proposed but they are still too compact to suffer from limited accuracy in handling local details and a long time of convergence. We contribute a neural simulation framework based on multi-resolution hash grid representation to introduce hierarchical consideration of global and local information, simultaneously. Furthermore, we propose two key strategies: 1) a numerical gradient method for computing high-order derivatives with boundary conditions; 2) a range analysis sample method for fast neural geometry boundary sampling with dynamic topologies. Our method shows much higher accuracy and strong flexibility for various simulation problems: e.g., large elastic deformations, complex fluid dynamics, and multi-scale phenomena which remain challenging for existing neural physical solvers.

NeurIPS Conference 2024 Conference Paper

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

• Tianbao Xie
• Danyang Zhang
• Jixuan Chen
• Xiaochuan Li
• Siheng Zhao
• Ruisheng Cao
• Toh J. Hua
• Zhoujun Cheng

Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72. 36% of the tasks, the best model achieves only 12. 24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at this https URL.

ICLR Conference 2024 Conference Paper

Shadow Cones: A Generalized Framework for Partial Order Embeddings

• Tao Yu
• Toni J. B. Liu
• Albert Tseng
• Christopher De Sa

Hyperbolic space has proven to be well-suited for capturing hierarchical relations in data, such as trees and directed acyclic graphs. Prior work introduced the concept of entailment cones, which uses partial orders defined by nested cones in the Poincar\'e ball to model hierarchies. Here, we introduce the ``shadow cones" framework, a physics-inspired entailment cone construction. Specifically, we model partial orders as subset relations between shadows formed by a light source and opaque objects in hyperbolic space. The shadow cones framework generalizes entailment cones to a broad class of formulations and hyperbolic space models beyond the Poincar\'e ball. This results in clear advantages over existing constructions: for example, shadow cones possess better optimization properties over constructions limited to the Poincar\'e ball. Our experiments on datasets of various sizes and hierarchical structures show that shadow cones consistently and significantly outperform existing entailment cone constructions. These results indicate that shadow cones are an effective way to model partial orders in hyperbolic space, offering physically intuitive and novel insights about the nature of such structures.

NeurIPS Conference 2024 Conference Paper

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

• Ruisheng Cao
• Fangyu Lei
• Haoyuan Wu
• Jixuan Chen
• Yeqiao Fu
• Hongcheng Gao
• Xinzhuang Xiong
• Hanchong Zhang

Data science and engineering workflows often span multiple stages, from warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As vision language models (VLMs) advance in multimodal understanding and code generation, VLM-based agents could potentially automate these workflows by generating SQL queries, Python code, and GUI operations. This automation can improve the productivity of experts while democratizing access to large-scale data analysis. In this paper, we introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks, derived from real-world use cases, evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems. To balance realistic simulation with evaluation simplicity, we devote significant effort to developing automatic configurations for task setup and carefully crafting evaluation metrics for each task. Furthermore, we supplement multimodal agents with comprehensive documents of these enterprise data software systems. Our empirical evaluation reveals that existing state-of-the-art LLM/VLM-based agents do not reliably automate full data workflows (14. 0% success). Even with step-by-step guidance, these agents still underperform in tasks that require fine-grained, knowledge-intensive GUI actions (16. 2%) and involve remote cloud-hosted workspaces (10. 6%). We hope that Spider2-V paves the way for autonomous multimodal agents to transform the automation of data science and engineering workflow. Our code and data are available at https: //spider2-v. github. io.

NeurIPS Conference 2024 Conference Paper

VLKEB: A Large Vision-Language Model Knowledge Editing Benchmark

• Han Huang
• Haitian Zhong
• Tao Yu
• Qiang Liu
• Shu Wu
• Liang Wang
• Tieniu Tan

Recently, knowledge editing on large language models (LLMs) has received considerable attention. Compared to this, editing Large Vision-Language Models (LVLMs) faces extra challenges from diverse data modalities and complicated model components, and data for LVLMs editing are limited. The existing LVLM editing benchmark, which comprises three metrics (Reliability, Locality, and Generality), falls short in the quality of synthesized evaluation images and cannot assess whether models apply edited knowledge in relevant content. Therefore, we employ more reliable data collection methods to construct a new Large $\textbf{V}$ision-$\textbf{L}$anguage Model $\textbf{K}$nowledge $\textbf{E}$diting $\textbf{B}$enchmark, $\textbf{VLKEB}$, and extend the Portability metric for more comprehensive evaluation. Leveraging a multi-modal knowledge graph, our image data are bound with knowledge entities. This can be further used to extract entity-related knowledge, which constitutes the base of editing data. We conduct experiments of different editing methods on five LVLMs, and thoroughly analyze how do they impact the models. The results reveal strengths and deficiencies of these methods and hopefully provide insights for future research. The codes and dataset are available at: https: //github. com/VLKEB/VLKEB.

TMLR Journal 2023 Journal Article

Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

• Aarohi Srivastava
• Abhinav Rastogi
• Abhishek Rao
• Abu Awal Md Shoeb
• Abubakar Abid
• Adam Fisch
• Adam R. Brown
• Adam Santoro

Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG- bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood develop- ment, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google- internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at a critical scale often involve multiple steps or components, or brittle metrics; social bias typically increases with scale in settings with ambiguous context, but this can be improved with prompting.

NeurIPS Conference 2023 Conference Paper

Coneheads: Hierarchy Aware Attention

• Albert Tseng
• Tao Yu
• Toni Liu
• Christopher M. De Sa

Attention networks such as transformers have achieved state-of-the-art performance in many domains. These networks rely heavily on the dot product attention operator, which computes the similarity between two points by taking their inner product. However, the inner product does not explicitly model the complex structural properties of real world datasets, such as hierarchies between data points. To remedy this, we introduce cone attention, a drop-in replacement for dot product attention based on hyperbolic entailment cones. Cone attention associates two points by the depth of their lowest common ancestor in a hierarchy defined by hyperbolic cones, which intuitively measures the divergence of two points and gives a $\textit{hierarchy aware}$ similarity score. We test cone attention on a wide variety of models and tasks and show that it improves task-level performance over dot product attention and other baselines, and is able to match dot-product attention with significantly fewer parameters. Our results suggest that cone attention is an effective way to capture hierarchical relationships when calculating attention.

EAAI Journal 2023 Journal Article

Corrigendum to “Self-supervised monocular depth estimation based on combining convolution and multilayer perceptron” [Eng. Appl. Artif. Intell. 117 (2023) 105587]

• Qiumei Zheng
• Tao Yu
• Fenghua Wang

YNIMG Journal 2023 Journal Article

Human anterior thalamic stimulation evoked cortical potentials align with intrinsic functional connectivity

• Di Wu
• Frederic L.W.V.J. Schaper
• Guangyuan Jin
• Lei Qi
• Jialin Du
• Xiaopeng Wang
• Yuke Wang
• Cuiping Xu

Characterizing human thalamocortical network is fundamental for understanding a vast array of human behaviors since the thalamus plays a central role in cortico-subcortical communication. Over the past few decades, advances in functional magnetic resonance imaging have allowed for spatial mapping of intrinsic resting-state functional connectivity (RSFC) between both cortical regions and in cortico-subcortical networks. Despite these advances, identifying the electrophysiological basis of human thalamocortical network architecture remains challenging. By leveraging stereoelectroencephalography electrodes temporarily implanted into distributed cortical regions and the anterior nucleus of the thalamus (ANT) of 10 patients with refractory focal epilepsy, we tested whether ANT stimulation evoked cortical potentials align with RSFC from the stimulation site, derived from a normative functional connectome (n = 1000). Our study identifies spatial convergence of ANT stimulation evoked cortical potentials and normative RSFC. Other than connections to the Papez circuit, the ANT was found to be closely connected to several distinct higher-order association cortices, including the precuneus, angular gyrus, dorsal lateral prefrontal cortex, and anterior insula. Remarkably, we found that the spatial distribution and magnitude of cortical-evoked responses to single-pulse electrical stimulation of the ANT aligned with the spatial pattern and strength of normative RSFC of the stimulation site. The present study provides electrophysiological evidence that stimulation evoked electrical activity flows along intrinsic brain networks connected on a thalamocortical level.

ICLR Conference 2023 Conference Paper

Random Laplacian Features for Learning with Hyperbolic Space

• Tao Yu
• Christopher De Sa

Due to its geometric properties, hyperbolic space can support high-fidelity embeddings of tree- and graph-structured data, upon which various hyperbolic networks have been developed. Existing hyperbolic networks encode geometric priors not only for the input, but also at every layer of the network. This approach involves repeatedly mapping to and from hyperbolic space, which makes these networks complicated to implement, computationally expensive to scale, and numerically unstable to train. In this paper, we propose a simpler approach: learn a hyperbolic embedding of the input, then map once from it to Euclidean space using a mapping that encodes geometric priors by respecting the isometries of hyperbolic space, and finish with a standard Euclidean network. The key insight is to use a random feature mapping via the eigenfunctions of the Laplace operator, which we show can approximate any isometry-invariant kernel on hyperbolic space. Our method can be used together with any graph neural networks: using even a linear graph model yields significant improvements in both efficiency and performance over other hyperbolic baselines in both transductive and inductive tasks.

EAAI Journal 2023 Journal Article

Self-supervised monocular depth estimation based on combining convolution and multilayer perceptron

• Qiumei Zheng
• Tao Yu
• Fenghua Wang

There are two mainstream approaches currently used for self-supervised monocular depth estimation. One option is to utilize a complete convolution method to construct the encoder and decoder; however, the local linear operation and the pooling method result in the loss of pixel information in each layer of the feature map, limiting the performance. Another way is to use the transformer and other methods for feature extraction on the encoder side, which are processed at a constant resolution at each stage and have a global receptive field. Therefore, more subtle depth features can be captured, and higher accuracy can be obtained. Unfortunately, the computational cost of self-attention is too large, which increases the memory overhead. With the comprehensive analysis of the advantages and disadvantages of the above two methods, this paper employs a combination of decomposed large kernel convolution and a multilayer perceptron (MLP) to design a new framework — CSMHNet (a hybrid of a Convolution, self-attention, and an MLP network). It cannot only compensate for the disadvantages of convolution static weights and locality but also significantly reduce the memory overhead compared to the transformer architecture while obtaining a more accurate and consistent depth. Experiments on the KITTI dataset demonstrate the effectiveness of our method, which significantly improves the depth prediction accuracy compared with other self-supervised methods.

NeurIPS Conference 2023 Conference Paper

Triangulation Residual Loss for Data-efficient 3D Pose Estimation

• Jiachen Zhao
• Tao Yu
• Liang An
• Yipeng Huang
• Fang Deng
• Qionghai Dai

This paper presents Triangulation Residual loss (TR loss) for multiview 3D pose estimation in a data-efficient manner. Existing 3D supervised models usually require large-scale 3D annotated datasets, but the amount of existing data is still insufficient to train supervised models to achieve ideal performance, especially for animal pose estimation. To employ unlabeled multiview data for training, previous epipolar-based consistency provides a self-supervised loss that considers only the local consistency in pairwise views, resulting in limited performance and heavy calculations. In contrast, TR loss enables self-supervision with global multiview geometric consistency. Starting from initial 2D keypoint estimates, the TR loss can fine-tune the corresponding 2D detector without 3D supervision by simply minimizing the smallest singular value of the triangulation matrix in an end-to-end fashion. Our method achieves the state-of-the-art 25. 8mm MPJPE and competitive 28. 7mm MPJPE with only 5\% 2D labeled training data on the Human3. 6M dataset. Experiments on animals such as mice demonstrate our TR loss's data-efficient training ability.

AIIM Journal 2022 Journal Article

An end-to-end tracking method for polyp detectors in colonoscopy videos

• Tao Yu
• Ne Lin
• Xu Zhang
• Yanqi Pan
• Huiyi Hu
• Wenfang Zheng
• Jiquan Liu
• Weiling Hu

Deep learning based computer-aided diagnosis technology demonstrates an encouraging performance in aspect of polyp lesion detection on reducing the miss rate of polyps during colonoscopies. However, to date, few studies have been conducted for tracking polyps that have been detected in colonoscopy videos, which is an essential and intuitive issue in clinical intelligent video analysis task (e. g. lesion counting, lesion retrieval, report generation). In the paradigm of conventional tracking-by-detection system, detection task for lesion localization is separated from the tracking task for cropped lesions re-identification. In the multi object tracking problem, each target is supposed to be tracked by invoking a tracker after the detector, which introduces multiple inferences and leads to external resource and time consumption. To tackle these problems, we proposed a plug-in module named instance tracking head (ITH) for synchronous polyp detection and tracking, which can be simply inserted into object detection frameworks. It embeds a feature-based polyp tracking procedure into the detector frameworks to achieve multi-task model training. ITH and detection head share the model backbone for low level feature extraction, and then low level feature flows into the separate branches for task-driven model training. For feature maps from the same receptive field, the region of interest head assigns these features to the detection head and the ITH, respectively, and outputs the object category, bounding box coordinates, and instance feature embedding simultaneously for each specific polyp target. We also proposed a method based on similarity metric learning. The method makes full use of the prior boxes in the object detector to provide richer and denser instance training pairs, to improve the performance of the model evaluation on the tracking task. Compared with advanced tracking-by-detection paradigm methods, detectors with proposed ITH can obtain comparative tracking performance but approximate 30% faster speed. Optimized model based on Scaled-YOLOv4 detector with ITH illustrates good trade-off between detection (mAP 91. 70%) and tracking (MOTA 92. 50% and Rank-1 Acc 88. 31%) task at the frame rate of 66 FPS. The proposed structure demonstrates the potential to aid clinicians in real-time detection with online tracking or offline retargeting of polyp instances during colonoscopies.

NeurIPS Conference 2022 Conference Paper

Mask-based Latent Reconstruction for Reinforcement Learning

• Tao Yu
• Zhizheng Zhang
• Cuiling Lan
• Yan Lu
• Zhibo Chen

For deep reinforcement learning (RL) from pixels, learning effective state representations is crucial for achieving high performance. However, in practice, limited experience and high-dimensional inputs prevent effective representation learning. To address this, motivated by the success of mask-based modeling in other research fields, we introduce mask-based reconstruction to promote state representation learning in RL. Specifically, we propose a simple yet effective self-supervised method, Mask-based Latent Reconstruction (MLR), to predict complete state representations in the latent space from the observations with spatially and temporally masked pixels. MLR enables better use of context information when learning state representations to make them more informative, which facilitates the training of RL agents. Extensive experiments show that our MLR significantly improves the sample efficiency in RL and outperforms the state-of-the-art sample-efficient RL methods on multiple continuous and discrete control benchmarks. Our code is available at https: //github. com/microsoft/Mask-based-Latent-Reconstruction.

NeurIPS Conference 2022 Conference Paper

Understanding Hyperdimensional Computing for Parallel Single-Pass Learning

• Tao Yu
• Yichi Zhang
• Zhiru Zhang
• Christopher M. De Sa

Hyperdimensional computing (HDC) is an emerging learning paradigm that computes with high dimensional binary vectors. There is an active line of research on HDC in the community of emerging hardware because of its energy efficiency and ultra-low latency---but HDC suffers from low model accuracy, with little theoretical understanding of what limits its performance. We propose a new theoretical analysis of the limits of HDC via a consideration of what similarity matrices can be expressed'' by binary vectors, and we show how the limits of HDC can be approached using random Fourier features (RFF). We extend our analysis to the more general class of vector symbolic architectures (VSA), which compute with high-dimensional vectors (hypervectors) that are not necessarily binary. We propose a new class of VSAs, finite group VSAs, which surpass the limits of HDC. Using representation theory, we characterize which similarity matrices can be expressed'' by finite group VSA hypervectors, and we show how these VSAs can be constructed. Experimental results show that our RFF method and group VSA can both outperform the state-of-the-art HDC model by up to 7. 6\% while maintaining hardware efficiency. This work aims to inspire a future interest on HDC in the ML community and connect to the hardware community.

ICRA Conference 2021 Conference Paper

Accurate and Robust Stereo Direct Visual Odometry for Agricultural Environment

• Tao Yu
• Junwei Zhou 0002
• Liangliang Wang
• Shengwu Xiong 0001

Vision-based localization and mapping in the agricultural environment is challenging due to the unstructured scene with unstable features, illumination variations, bumpy roads, and dynamic environmental objects. To address these challenges, we propose an accurate and robust stereo direct visual odometry system with modifications on Stereo-DSO. We firstly select some well-matched static stereo points in the latest keyframe to improve the accuracy of inverse depth calculation for tracking. The inverse depth can further distinguish close objects from background, which will avoid large and far-away scene objects in keyframe determination. To boost efficiency and accuracy at the tracking stage, we propose a point selection method to sample map points and remove outliers. Furthermore, altitude smoothness verification with a local flat ground assumption and recovery method for tracking failure on bumpy roads are proposed to improve the system’s robustness. Finally, a far-away keyframe is reserved in the sliding window to alleviate the orientation drift since the agricultural robots usually move straightly following the crop row. Our system achieved new state-of-the-art results on Flourish dataset and the recently released Rosario dataset.

EAAI Journal 2021 Journal Article

Emergency fault affected wide-area automatic generation control via large-scale deep reinforcement learning

• Jiawen Li
• Tao Yu
• Xiaoshun Zhang

When a complex large power system is in an emergency, the conventional discrete emergency control strategy implemented will cause excess load or derivative accidents like line power overload, thereby raising the operation risk. To overcome the problems of excessive/insufficient regulation and subsequent accidents in the above context, this paper proposes a wide area automatic generation control (WA-AGC) framework, which integrates the emergency control strategy and a performance-based frequency regulation market mechanism. According to the frequency status of the power system, WA-AGC divides the AGC into four intervals, emergency AGC (EAGC), conventional AGC (CAGC), AGC transition and optimal power flow (OPF). These four together realize a comprehensive optimization of frequency and system stability as well as economy Based on the above framework, a swarm agent exploration distributed multiple delayed deep policy gradient algorithm (SAE-MD3) is developed, which uses multiple explorers with different exploration strategies for distributed optimization. In addition, several technologies are introduced to prevent Q value overestimation and generate a more robust optimal AGC strategy. Afterward, the effectiveness and feasibility of WA-AGC are verified through the simulations of an IEEE-9 two-area system and an IEEE-118 two-area system. Compared to conventional AGC strategies, the WA-AGC algorithm reduces the constraint violation time of the power line by 92. 06% and the power generation cost by 0. 27% as well as improves the CPS1 index by 0. 04%.

YNIMG Journal 2021 Journal Article

Functional mapping of language-related areas from natural, narrative speech during awake craniotomy surgery

• Tianyi Zhou
• Tao Yu
• Zheng Li
• Xiaoxia Zhou
• Jianbin Wen
• Xiaoli Li

Accurate localization of brain regions responsible for language and cognitive functions in epilepsy patients is important. Electrocorticography (ECoG)-based real-time functional mapping (RTFM) has been shown to be a safer alternative to electrical cortical stimulation mapping (ESM), which is currently the clinical/gold standard. Conventional methods for analyzing RTFM data mostly account for the ECoG signal in certain frequency bands, especially high gamma. Compared to ESM, they have limited accuracy when assessing channel responses. In the present study, we developed a novel RTFM method based on tensor component analysis (TCA) to address the limitations of current estimation methods. Our approach analyzes the whole frequency spectrum of the ECoG signal during natural continuous speech. We construct third-order tensors that contain multichannel time-frequency information and use TCA to extract low-dimensional temporal, spectral and spatial modes. Temporal modulation scores (correlation values) are then calculated between the time series of voice envelope features and TCA-estimated temporal courses, and significant temporal modulation determines which components' channel weightings are displayed to the neurosurgeon as a guide for follow-up ESM. In our experiments, data from thirteen patients with refractory epilepsy were recorded during preoperative evaluation for their epileptogenic zones (EZs), which were located adjacent to the eloquent cortex. Our results showed higher detection accuracy of our proposed method in a narrative speech task, suggesting that our method complements ESM and is an improvement over the prior RTFM method. To our knowledge, this is the first TCA-based method to pinpoint language-specific brain regions during continuous speech that uses whole-band ECoG.

AAAI Conference 2021 Conference Paper

Learning Omni-Frequency Region-adaptive Representations for Real Image Super-Resolution

• Xin Li
• Xin Jin
• Tao Yu
• Simeng Sun
• Yingxue Pang
• Zhizheng Zhang
• Zhibo Chen

Traditional single image super-resolution (SISR) methods that focus on solving single and uniform degradation (i. e. , bicubic down-sampling), typically suffer from poor performance when applied into real-world low-resolution (LR) images due to the complicated realistic degradations. The key to solving this more challenging real image super-resolution (RealSR) problem lies in learning feature representations that are both informative and content-aware. In this paper, we propose an Omni-frequency Region-adaptive Network (OR- Net) to address both challenges, here we call features of all low, middle and high frequencies omni-frequency features. Specifically, we start from the frequency perspective and design a Frequency Decomposition (FD) module to separate different frequency components to comprehensively compensate the information lost for real LR image. Then, considering the different regions of real LR image have different frequency information lost, we further design a Region-adaptive Frequency Aggregation (RFA) module by leveraging dynamic convolution and spatial attention to adaptively restore frequency components for different regions. The extensive experiments endorse the effective, and scenario-agnostic nature of our OR-Net for RealSR.

NeurIPS Conference 2021 Conference Paper

PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning

• Tao Yu
• Cuiling Lan
• Wenjun Zeng
• Mingxiao Feng
• Zhizheng Zhang
• Zhibo Chen

Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i. e. , state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose a novel method, dubbed PlayVirtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, PlayVirtual predicts future states in a latent space based on the current state and action by a dynamics model and then predicts the previous states by a backward dynamics model, which forms a trajectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce a trajectory to meet the cycle consistency constraint, which can significantly enhance the data efficiency. We validate the effectiveness of our designs on the Atari and DeepMind Control Suite benchmarks. Our method achieves the state-of-the-art performance on both benchmarks. Our code is available at https: //github. com/microsoft/Playvirtual.

NeurIPS Conference 2021 Conference Paper

Representing Hyperbolic Space Accurately using Multi-Component Floats

• Tao Yu
• Christopher M. De Sa

Hyperbolic space is particularly useful for embedding data with hierarchical structure; however, representing hyperbolic space with ordinary floating-point numbers greatly affects the performance due to its \emph{ineluctable} numerical errors. Simply increasing the precision of floats fails to solve the problem and incurs a high computation cost for simulating greater-than-double-precision floats on hardware such as GPUs, which does not support them. In this paper, we propose a simple, feasible-on-GPUs, and easy-to-understand solution for numerically accurate learning on hyperbolic space. We do this with a new approach to represent hyperbolic space using multi-component floating-point (MCF) in the Poincar{\'e} upper-half space model. Theoretically and experimentally we show our model has small numerical error, and on embedding tasks across various datasets, models represented by multi-component floating-points gain more capacity and run significantly faster on GPUs than prior work.

AAAI Conference 2020 Conference Paper

Region Normalization for Image Inpainting

• Tao Yu
• Zongyu Guo
• Xin Jin
• Shilin Wu
• Zhibo Chen
• Weiping Li
• Zhizheng Zhang
• Sen Liu

Feature Normalization (FN) is an important technique to help neural network training, which typically normalizes features across spatial dimensions. Most previous image inpainting methods apply FN in their networks without considering the impact of the corrupted regions of the input image on normalization, e. g. mean and variance shifts. In this work, we show that the mean and variance shifts caused by full-spatial FN limit the image inpainting network training and we propose a spatial region-wise normalization named Region Normalization (RN) to overcome the limitation. RN divides spatial pixels into different regions according to the input mask, and computes the mean and variance in each region for normalization. We develop two kinds of RN for our image inpainting network: (1) Basic RN (RN-B), which normalizes pixels from the corrupted and uncorrupted regions separately based on the original inpainting mask to solve the mean and variance shift problem; (2) Learnable RN (RN-L), which automatically detects potentially corrupted and uncorrupted regions for separate normalization, and performs global affine transformation to enhance their fusion. We apply RN-B in the early layers and RN-L in the latter layers of the network respectively. Experiments show that our method outperforms current state-ofthe-art methods quantitatively and qualitatively. We further generalize RN to other inpainting networks and achieve consistent performance improvements.

NeurIPS Conference 2019 Conference Paper

A New Defense Against Adversarial Images: Turning a Weakness into a Strength

• Shengyuan Hu
• Tao Yu
• Chuan Guo
• Wei-Lun Chao
• Kilian Weinberger

Natural images are virtually surrounded by low-density misclassified regions that can be efficiently discovered by gradient-guided search --- enabling the generation of adversarial images. While many techniques for detecting these attacks have been proposed, they are easily bypassed when the adversary has full knowledge of the detection mechanism and adapts the attack strategy accordingly. In this paper, we adopt a novel perspective and regard the omnipresence of adversarial perturbations as a strength rather than a weakness. We postulate that if an image has been tampered with, these adversarial directions either become harder to find with gradient methods or have substantially higher density than for natural images. We develop a practical test for this signature characteristic to successfully detect adversarial attacks, achieving unprecedented accuracy under the white-box setting where the adversary is given full knowledge of our detection mechanism.

NeurIPS Conference 2019 Conference Paper

Numerically Accurate Hyperbolic Embeddings Using Tiling-Based Models

• Tao Yu
• Christopher De Sa

Hyperbolic embeddings achieve excellent performance when embedding hierarchical data structures like synonym or type hierarchies, but they can be limited by numerical error when ordinary floating-point numbers are used to represent points in hyperbolic space. Standard models such as the Poincar{\'e} disk and the Lorentz model have unbounded numerical error as points get far from the origin. To address this, we propose a new model which uses an integer-based tiling to represent \emph{any} point in hyperbolic space with provably bounded numerical error. This allows us to learn high-precision embeddings without using BigFloats, and enables us to store the resulting embeddings with fewer bits. We evaluate our tiling-based model empirically, and show that it can both compress hyperbolic embeddings (down to $2\%$ of a Poincar{\'e} embedding on WordNet Nouns) and learn more accurate embeddings on real-world datasets.

IS Journal 2019 Journal Article

Parallel Cyber-Physical-Social Systems Based Smart Energy Robotic Dispatcher and Knowledge Automation: Concepts, Architectures, and Challenges

• Lefeng Cheng
• Tao Yu
• Xiaoshun Zhang
• Bo Yang

We propose a novel concept of the robot of energy control based on parallel cyber-physical-social systems for the next generation of energy and electric power systems. We thoroughly investigate its knowledge automation technologies and discuss its main challenges in preparation of promising evolution from Energy 4. 0 to Energy 5. 0 in China.

ICML Conference 2019 Conference Paper

Simplifying Graph Convolutional Networks

• Felix Wu
• Amauri H. Souza Jr.
• Tianyi Zhang 0007
• Christopher Fifty
• Tao Yu
• Kilian Q. Weinberger

Graph Convolutional Networks (GCNs) and their variants have experienced significant attention and have become the de facto methods for learning graph representations. GCNs derive inspiration primarily from recent deep learning approaches, and as a result, may inherit unnecessary complexity and redundant computation. In this paper, we reduce this excess complexity through successively removing nonlinearities and collapsing weight matrices between consecutive layers. We theoretically analyze the resulting linear model and show that it corresponds to a fixed low-pass filter followed by a linear classifier. Notably, our experimental evaluation demonstrates that these simplifications do not negatively impact accuracy in many downstream applications. Moreover, the resulting model scales to larger datasets, is naturally interpretable, and yields up to two orders of magnitude speedup over FastGCN.

YNICL Journal 2018 Journal Article

Gray-matter-specific MR imaging improves the detection of epileptogenic zones in focal cortical dysplasia: A new sequence called fluid and white matter suppression (FLAWS)

• Xin Chen
• Tianyi Qian
• Tobias Kober
• Guojun Zhang
• Zhiwei Ren
• Tao Yu
• Yueshan Piao
• Nan Chen

Objectives: To evaluate the diagnostic value and characteristic features of FCD epileptogenic zones using a novel sequence called fluid and white matter suppression (FLAWS). Materials and methods: Thirty-nine patients with pathologically confirmed FCD and good surgery outcomes (class I or II, according to the Engel Epilepsy Surgery Outcome Scale) were retrospectively included in the study. All the patients underwent a preoperative whole-brain MRI examination that included conventional sequences (T2WI, T1WI, two-dimensional (2D) axial, coronal fluid-attenuated inversion recovery [FLAIR]) and FLAWS. An additional 3D-FLAIR MRI sequence was performed in 17 patients. To evaluate the sensitivity and specificity of FLAWS and investigate the cause of false-positives, 36 healthy volunteers were recruited as normal controls. Two radiologists evaluated all the image data. The detection rates of the FCD epileptogenic zone on different sequences were compared based on five criteria: abnormal cortical morphology (thickening, thinning, or abnormally deep sulcus); abnormal cortical signal intensity; blurred gray-white matter junction; abnormal signal intensity of the subcortical white matter, and the transmantle sign. The sensitivity and specificity of FLAWS for detecting the FCD lesions were calculated with the reviewers blinded to all the clinical information, i.e. to the patient identity and the location of the resected regions. To explore how many features were sufficient for the diagnosis of the epileptogenic zones, the frequency of each criterion in the resected regions and their combinations were assessed on FLAWS, according to the results of the assessment when the reviewers were aware of the location of the resected regions. Based on the findings of the 17 patients with an additional 3D-FLAIR scan when the reviewers were aware of the location of the resected regions, quantitative analysis of the regions of interest was used to compare the tissue contrast among 2D-axial FLAIR, 3D-FLAIR, and the FLAWS sequence. Visualization score analysis was used to evaluate the visualization of the five features on conventional, 3D-FLAIR, and FLAWS images. Finally, to explore the reason for false-positive results, a further evaluation of the whole brain FLAWS images was conducted for all the subjects. Results: The sensitivity and specificity for detecting the FCD lesions on the FLAWS sequence were 71.9% and 71.1%, respectively. When the reviewers were blinded to the location of the resected regions, the detection rate of the FLAWS sequence was significantly higher than that of the conventional sequences (P = 0.00). In the 17 patients who underwent an additional 3D FLAIR scan, no statistically significant difference was found between the FLAWS and the 3D-FLAIR (P = 0.25). All the patients had at least two imaging features, one of which was "the blurred junction of the gray-white matter." The transmantle sign, which is widely believed to be a specific feature of FCD type II, could also be observed in type I on the FLAWS sequence. The relative tissue contrast of FLAWS was higher than that of the 2D-FLAIR with respect to lesion/white matter (WM), deep gray matter (GM)/WM, and cortex/WM (P = 0.00 for all three measures) and higher than that of the 3D-FLAIR with respect to the lesion/WM (P = 0.01). The visualization score analysis showed that the visualization of FLAWS was more enhanced than that of the conventional and 3D-FLAIR images with respect to the blurred junction (P = 0.00 for both comparisons) and the abnormal signal intensity of the subcortical white matter (P = 0.01 for both comparisons). The thin-threadlike signal and individual FCD features outside the epileptogenic regions were considered the primary cause of the false-positive results of FLAWS. Conclusions: FLAWS can help in the detection of FCD epileptogenic zones. It is recommended that epileptogenic zone on FLAWS be diagnosed based on a combination of two features, one of which should be the "blurred junction of the gray-white matter" in types I and II. In type III, the combination of "the blurred junction of the gray-white matter" with "abnormal signal intensity of subcortical white matter" is recommended.

EAAI Journal 2007 Journal Article

Integrating relevance vector machines and genetic algorithms for optimization of seed-separating process

• Jin Yuan
• Kesheng Wang
• Tao Yu
• Minglung Fang

A hybrid intelligent approach based on relevance vector machines (RVMs) and genetic algorithms (GAs) has been developed for optimal control of parameters of nonlinear manufacturing processes. It concerns the finding of the near-optimal control parameters of the nonlinear discrete manufacturing process with a specific objective. First, the nonlinear process with measurement noise is regressed by the relevance vector learning mechanism based on a kernel-based Bayesian framework. For minimizing the approximate error, uniform design sampling, online incremental learning and cross-validation are used in the learning process of RVMs. Such well-trained models become a specialized process simulation tool, which is valuable in prediction and optimization of nonlinear processes. Next, the near-optimal setpoints of the control system, which maximize the objective function, are sought by GAs from the numerous values of the objective function obtained from the simulation. As a case study, the seed separator system (5XZW-1. 5) is used for evaluating the proposed intelligent approach. The control parameters to reach the maximum weighted objective, which combine the system output and evaluation functions, are optimized. The experimental results show the effectiveness of the proposed hybrid approach.