Author name cluster

Ziwei Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

22 papers

2 author rows

AAAI Conference 2026 Conference Paper

10 Open Challenges Steering the Future of Vision-Language-Action Models

Soujanya Poria
Navonil Majumder
Chia-Yu Hung
Amir Ali Bagherzadeh
Chuan Li
Kenneth Kwok
Ziwei Wang
Cheston Tan

Due to their ability of follow natural language instructions, vision-language-action (VLA) models are increasingly preva- lent in the embodied AI arena, following the widespread suc- cess of their precursors—LLMs and VLMs. In this paper, we discuss 10 principal milestones in the ongoing develop- ment of VLA models—multimodality, reasoning, data, eval- uation, cross-robkot action generalization, efficiency, whole- body coordination, safety, agents, and coordination with hu- mans. Furthermore, we discuss the emerging trends of us- ing spatial understanding, modeling world dynamics, post training, and data synthesis—all aiming to reach these mile- stones. Through these discussions, we hope to bring attention to the research avenues that may accelerate the development of VLA models into wider acceptability.

PDF Details DOI

JBHI Journal 2026 Journal Article

DBConformer: Dual-Branch Convolutional Transformer for EEG Decoding

Ziwei Wang
Hongbin Wang
Tianwang Jia
Xingyi He
Siyang Li
Dongrui Wu

Electroencephalography (EEG)-based brain-computer interfaces (BCIs) transform spontaneous/evoked neural activity into control commands for external communication. While convolutional neural networks (CNNs) remain the mainstream backbone for EEG decoding, their inherently short receptive field makes it difficult to capture long-range temporal dependencies and global inter-channel relationships. Recent CNN-Transformer (Conformer) hybrids partially address this issue, but most adopt a serial design, resulting in suboptimal integration of local and global features, and often overlook explicit channel-wise modeling. To address these limitations, we propose DBConformer, a dual-branch convolutional Transformer network tailored for EEG decoding. It integrates a temporal Conformer to model long-range temporal dependencies and a spatial Conformer to extract inter-channel interactions, capturing both temporal dynamics and spatial patterns in EEG signals. A lightweight channel attention module further refines spatial representations by assigning data-driven importance to EEG channels. Extensive experiments under four evaluation settings on three paradigms, including motor imagery, seizure detection, and steady-state visual evoked potential, demonstrated that DBConformer consistently outperformed 13 competitive baseline models, with over an eight-fold reduction in parameters than current high-capacity EEG Conformer architecture. Furthermore, the visualization results confirmed that the features extracted by DBConformer are physiologically interpretable and aligned with prior knowledge. The superior performance and interpretability of DBConformer make it reliable for accurate, robust, and explainable EEG decoding.

Details DOI

AAAI Conference 2026 Conference Paper

History-Aware Reasoning for GUI Agents

Ziwei Wang
Leyang Yang
Xiaoxuan Tang
Sheng Zhou
Dajun Chen
Wei Jiang
Yong Li

Advances in Multimodal Large Language Models have significantly enhanced Graphical User Interface (GUI) automation. Equipping GUI agents with reliable episodic reasoning capabilities is essential for bridging the gap between users’ concise task descriptions and the complexities of real-world execution. Current methods integrate Reinforcement Learning (RL) with System-2 Chain-of-Thought, yielding notable gains in reasoning enhancement. For long-horizon GUI tasks, historical interactions connect each screen to the goal-oriented episode chain, and effectively leveraging these clues is crucial for the current decision. However, existing native GUI agents exhibit weak short-term memory in their explicit reasoning, interpreting the chained interactions as discrete screen understanding, i.e., unawareness of the historical interactions within the episode. This history-agnostic reasoning challenges their performance in GUI automation. To alleviate this weakness, we propose a History-Aware Reasoning (HAR) framework, which encourages an agent to reflect on its own errors and acquire episodic reasoning knowledge from them via tailored strategies that enhance short-term memory in long-horizon interaction. The framework mainly comprises constructing a reflective learning scenario, synthesizing tailored correction guidelines, and designing a hybrid RL reward function. Using the HAR framework, we develop a native end-to-end model, HAR-GUI-3B, which alters the inherent reasoning mode from history-agnostic to history-aware, equipping the GUI agent with stable short-term memory and reliable perception of screen details. Comprehensive evaluations across a range of GUI-related benchmarks demonstrate the effectiveness and generalization of our method.

PDF Details DOI

JBHI Journal 2026 Journal Article

LineGRN: A Line Graph Neural Network for Gene Regulatory Network Inference

Ziwei Wang
Ge Xu
Weiming Yu
Le Ou-Yang

Gene regulatory networks (GRNs) depicts the complex interactions between transcription factors and target genes, offering profound insight into deciphering the mechanisms of cellular processes. The advancement of single-cell RNA sequencing (scRNA-seq) technologies has provided a crucial perspective for inferring GRNs at single-cell resolution, leading to the development of numerous computational methods for GRN inference. However, most existing methods fail to adequately capture the association patterns between gene pairs, and the low-degree-node-dominated topology of prior GRNs imposes fundamental limitations on information propagation. In this study, we propose LineGRN, a novel line graph neural network framework for inferring GRNs from scRNA-seq data. By modeling the neighborhood relationships between gene pairs, LineGRN effectively preserves interaction signals within the topological structure. Moreover, the line graph transformation produces a high-degree-node-dominated local network topology, which enables more efficient information propagation. Comprehensive experiments on real datasets demonstrate that LineGRN significantly outperforms seven state-of-the-art methods. Furthermore, LineGRN exhibits low sensitivity to parameter variations and noise interference. Notably, case studies provide empirical evidence of the model’s ability to uncover potential TF-target regulatory associations.

Details DOI

EAAI Journal 2026 Journal Article

Physics-integrated intelligent method for propeller aerodynamic property predictions of electric aircraft

Wei Zhang
Ziwei Wang
Xiang Li
Song Xiang

Aerodynamic properties including thrust and torque are crucial parameters for propellers. Accurate estimations of the aerodynamic properties are of great importance for propeller's design. This paper proposes a physics-integrated intelligent method to estimate the thrust and torque of propellers. First, wind tunnel tests are conducted to collect experimental data at various wind and rotational speeds. Theoretical data is simulated using the fundamental Strip Theory, and the experimental data is augmented with the theoretical data. Next, a deep neural network is developed that integrates both experimental and simulation data. To further optimize the model and reduce reliance on the practical wind tunnel data, the impact of the key parameters on the performance of the deep neural networks is examined. This physics-integrated intelligent model is shown to be effective in predicting aerodynamic parameters for propellers, significantly decreasing the need for large amounts of real wind tunnel data. This method offers a promising solution for designing electric aircraft propellers while minimizing experimental costs compared to other high-fidelity methods.

Details DOI

AAAI Conference 2026 Conference Paper

ProBench: Benchmarking GUI Agents with Accurate Process Information

Leyang Yang
Ziwei Wang
Xiaoxuan Tang
Sheng Zhou
Dajun Chen
Wei Jiang
Yong Li

With the deep integration of artificial intelligence and interactive technology, Graphical User Interface (GUI) Agent, as the carrier connecting goal-oriented natural language and real-world devices, has received widespread attention from the community. Contemporary benchmarks aim to evaluate the comprehensive capabilities of GUI agents in GUI operation tasks, generally determining task completion solely by inspecting the final screen state. However, GUI operation tasks consist of multiple chained steps while not all critical information is presented in the final few pages. Although a few research has begun to incorporate intermediate steps into evaluation, accurately and automatically capturing this process information still remains an open challenge. To address this weakness, we introduce ProBench, a comprehensive mobile benchmark with over 200 challenging GUI tasks covering widely-used scenarios. Remaining the traditional State-related Task evaluation, we extend our dataset to include Process-related Task and design a specialized evaluation method. A newly introduced Process Provider automatically supplies accurate process information, enabling presice assessment of agent's performance. Our evaluation of advanced GUI agents reveals significant limitations for real-world GUI scenarios. These shortcomings are prevalent across diverse models, including both large-scale generalist models and smaller, GUI-specific models. A detailed error analysis further exposes several universal problems, outlining concrete directions for future improvements.

PDF Details DOI

AAAI Conference 2026 Conference Paper

SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheets

Ziwei Wang
Jiayuan Su
Mengyu Zhou
Huaxing Zeng
Mengni Jia
Xiao Lv
Haoyu Dong
Xiaojun Ma

Understanding and reasoning over complex spreadsheets remain fundamental challenges for large language models (LLMs), which often struggle with intricate structures and rely solely on neural computation. In this work, we propose SheetBrain, a neuro-symbolic dual-workflow agent framework for precise and interpretable reasoning over tabular data. SheetBrain consists of an understanding module that produces a comprehensive overview of the spreadsheet, including structural summaries and query-specific analyses to guide execution; an execution module that integrates a Python sandbox with preloaded table-processing libraries and an Excel helper toolkit for effective data manipulation; and a validation module that verifies the correctness of reasoning and answers, triggering re-execution if necessary. We evaluate SheetBrain on multiple public QA and manipulation benchmarks, and introduce SheetBench, a new benchmark targeting large, multi-table, and structurally complex spreadsheets. Experimental results show that SheetBrain significantly improves reasoning performance on both existing benchmarks and the more challenging scenarios presented in SheetBench.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Towards Scalable Web Accessibility Audit with MLLMs as Copilots

Ming Gu
Ziwei Wang
Sicen Lai
Zirui Gao
Sheng Zhou
Jiajun Bu

Ensuring web accessibility is crucial for advancing social welfare, justice, and equality in digital spaces, yet the vast majority of website user interfaces remain non-compliant, due in part to the resource-intensive and unscalable nature of current auditing practices. While WCAG-EM offers a structured methodology for site-wise conformance evaluation, it involves great human efforts and lacks practical support for execution at scale. In this work, we present an auditing framework, AAA, which operationalizes WCAG-EM through a human-AI partnership model. AAA is anchored by two key innovations: GRASP, a graph-based multimodal sampling method that ensures representative page coverage via learned embeddings of visual, textual, and relational cues; and MaC, a multimodal large language model-based copilot strategy that supports auditors through cross-modal reasoning and intelligent assistance in high-effort tasks. Together, these components enable scalable, end-to-end web accessibility auditing, empowering human auditors with AI-enhanced assistance for real-world impact. We further contribute four novel datasets designed for benchmarking core stages of the audit pipeline. Extensive experiments demonstrate the effectiveness of our methods, providing insights that small-scale language models can serve as capable experts when fine-tuned.

PDF Details DOI

JBHI Journal 2026 Journal Article

TransGRN: A Transfer Learning-Based Framework for Inferring Gene Regulatory Networks Across Cell Lines

Ge Xu
Ziwei Wang
Jiahao Zhou
Weiming Yu
Le Ou-Yang

Inferring gene regulatory networks (GRNs) is critical for understanding the mechanisms that govern cellular behavior. Advances in single-cell RNA sequencing (scRNA-seq) have enabled GRN analysis at single-cell resolution and stimulated the development of many computational methods. However, most existing approaches depend heavily on extensive prior regulatory information, which limits their effectiveness in few-shot settings where such data for the target cell line are scarce or unavailable. To address this challenge, we propose TransGRN, a transfer learning–based method for inferring gene regulatory networks (GRNs) across cell lines. TransGRN adopts a cross-cell-line pre-training strategy that combines scRNA-seq data from multiple source cell lines with biological knowledge obtained from large language models. In addition, it includes a regulatory interaction extraction module that integrates gene expression profiles with semantic information. By transferring generalizable gene–gene regulatory patterns from source to target cell lines, TransGRN achieves state-of-the-art performance in both benchmark tests and few-shot GRN inference tasks.

Details DOI

IROS Conference 2025 Conference Paper

Adaptive Motion Scaling in Teleoperated Robotic Surgery based on Human Intention and Attention

Yiming Zhai
Jingsong Liu
Yating Luo
Ziwei Wang
Yao Guo 0002

In teleoperated surgery, the motion scaling factor directly influences both the operator’s control precision of surgical instruments and operational comfort. Previous studies have revealed that the master manipulator state and operator’s gaze information can reflect the complexity of surgical operations and the operator’s intention to some extent. Although enabling real-time adjustment of scaling factors, they were limited by the narrow range of core parameters and the results were significantly influenced by subjective factors. To tackle these challenges, this paper presents a multi-dimensional adaptive motion scaling strategy based on the Bayesian optimization. The prediction of operator’s intention and attention is achieved by integrating multiple dimensional parameters, including master-slave manipulator states, gaze information, as well as pupillary data, all of which have been experimentally validated. Specifically, there exists a significant temporal synchronization between the Index of Pupillary Activity (IPA) and teleoperation tasks, which aligns with research on the correlation between IPA and attention levels. Furthermore, to evaluate the proposed adaptive scaling strategy, we combine subjective questionnaire surveys with objective metric assessments, effectively reducing the excessive influence of operators’ personal conditions and proficiency levels on optimization results.

Details

ICLR Conference 2025 Conference Paper

EmbodiedSAM: Online Segment Any 3D Thing in Real Time

Xiuwei Xu
Huangxing Chen
Linqing Zhao
Ziwei Wang
Jie Zhou 0001
Jiwen Lu

Embodied tasks require the agent to fully understand 3D scenes simultaneously with its exploration, so an online, real-time, fine-grained and highly-generalized 3D perception model is desperately needed. Since high-quality 3D data is limited, directly training such a model in 3D is infeasible. Meanwhile, vision foundation models (VFM) has revolutionized the field of 2D computer vision with superior performance, which makes the use of VFM to assist embodied 3D perception a promising direction. However, most existing VFM-assisted 3D perception methods are either offline or too slow that cannot be applied in practical embodied tasks. In this paper, we aim to leverage Segment Anything Model (SAM) for real-time 3D instance segmentation in an online setting. This is a challenging problem since future frames are not available in the input streaming RGB-D video, and an instance may be observed in several frames so efficient object matching between frames is required. To address these challenges, we first propose a geometric-aware query lifting module to represent the 2D masks generated by SAM by 3D-aware queries, which is then iteratively refined by a dual-level query decoder. In this way, the 2D masks are transferred to fine-grained shapes on 3D point clouds. Benefit from the query representation for 3D masks, we can compute the similarity matrix between the 3D masks from different views by efficient matrix operation, which enables real-time inference. Experiments on ScanNet, ScanNet200, SceneNN and 3RScan show our method achieves state-of-the-art performance among online 3D perception models, even outperforming offline VFM-assisted 3D instance segmentation methods by a large margin. Our method also demonstrates great generalization ability in several zero-shot dataset transferring experiments and show great potential in data-efficient setting.

Details

IROS Conference 2025 Conference Paper

ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model

Tengbo Yu
Guanxing Lu
Zaijia Yang
Haoyuan Deng
Season Si Chen
Jiwen Lu
Wenbo Ding 0001
Guoqiang Hu

Multi-task robotic bimanual manipulation is becoming increasingly popular as it enables sophisticated tasks that require diverse dual-arm collaboration patterns. Compared to unimanual manipulation, bimanual tasks pose challenges to understanding the multi-body spatiotemporal dynamics. An existing method ManiGaussian [30] pioneers encoding the spatiotemporal dynamics into the visual representation via Gaussian world model for single-arm settings, which ignores the interaction of multiple embodiments for dual-arm systems with significant performance drop. In this paper, we propose ManiGaussian++, an extension of ManiGaussian framework that improves multi-task bimanual manipulation by digesting multi-body scene dynamics through a hierarchical Gaussian world model. To be specific, we first generate task-oriented Gaussian Splatting from intermediate visual features, which aims to differentiate acting and stabilizing arms for multi-body spatiotemporal dynamics modeling. We then build a hierarchical Gaussian world model with the leader-follower architecture, where the multi-body spatiotemporal dynamics is mined for intermediate visual representation via future scene prediction. The leader predicts Gaussian Splatting deformation caused by motions of the stabilizing arm, through which the follower generates the physical consequences resulted from the movement of the acting arm. As a result, our method significantly outperforms the current state-of-the-art bimanual manipulation techniques by an improvement of 20. 2% in 10 simulated tasks, and achieves 60% success rate on average in 9 challenging real-world tasks. Our code is available at https://github.com/April-Yz/ManiGaussian_Bimanual.

Details

NeurIPS Conference 2025 Conference Paper

TGA: True-to-Geometry Avatar Dynamic Reconstruction

Bo Guo
Sijia Wen
Ziwei Wang
Yifan Zhao

Recent advances in 3D Gaussian Splatting (3DGS) have improved the visual fidelity of dynamic avatar reconstruction. However, existing methods often overlook the inherent chromatic similarity of human skin tones, leading to poor capture of intricate facial geometry under subtle appearance changes. This is caused by the affine approximation of Gaussian projection, which fails to be perspective-aware to depth-induced shear effects. To this end, we propose True-to-Geometry Avatar Dynamic Reconstruction (TGA), a perspective-aware 4D Gaussian avatar framework that sensitively captures fine-grained facial variations for accurate 3D geometry reconstruction. Specifically, to enable color-sensitive and geometry-consistent Gaussian representations under dynamic conditions, we introduce Perspective-Aware Gaussian Transformation that jointly models temporal deformations and spatial projection by integrating Jacobian-guided adaptive deformation into the homogeneous formulation. Furthermore, we develop Incremental BVH Tree Pivoting to enable fast frame-by-frame mesh extraction for 4D Gaussian representations. A dynamic Gaussian Bounding Volume Hierarchy (BVH) tree is used to model the topological relationships among points, where active ones are filtered out by BVH pivoting and subsequently re-triangulated for surface reconstruction. Extensive experiments demonstrate that TGA achieves superior geometric accuracy.

PDF Details

ICLR Conference 2025 Conference Paper

ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

Guanxing Lu
Ziwei Wang
Changliu Liu
Jiwen Lu
Yansong Tang

Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects and the related Bayesian uncertainty for close-loop planning. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency. Project page: https://guanxinglu.github.io/thinkbot/.

Details

NeurIPS Conference 2024 Conference Paper

Does Egalitarian Fairness Lead to Instability? The Fairness Bounds in Stable Federated Learning Under Altruistic Behaviors

Jiashi Gao
Ziwei Wang
Xiangyu Zhao
Xin Yao
Xuetao Wei

Federated learning (FL) offers a machine learning paradigm that protects privacy, allowing multiple clients to collaboratively train a global model while only accessing their local data. Recent research in FL has increasingly focused on improving the uniformity of model performance across clients, a fairness principle known as egalitarian fairness. However, achieving egalitarian fairness in FL may sacrifice the model performance for data-rich clients to benefit those with less data. This trade-off raises concerns about the stability of FL, as data-rich clients may opt to leave the current coalition and join another that is more closely aligned with its expected high performance. In this context, our work rigorously addresses the critical concern: Does egalitarian fairness lead to instability? Drawing from game theory and social choice theory, we initially characterize fair FL systems as altruism coalition formation games (ACFGs) and reveal that the instability issues emerging from the pursuit of egalitarian fairness are significantly related to the clients’ altruism within the coalition and the configuration of the friends-relationship networks among the clients. Then, we theoretically propose the optimal egalitarian fairness bounds that an FL coalition can achieve while maintaining core stability under various types of altruistic behaviors. The theoretical contributions clarify the quantitative relationships between achievable egalitarian fairness and the disparities in the sizes of local datasets, disproving the misconception that egalitarian fairness inevitably leads to instability. Finally, we conduct experiments to evaluate the consistency of our theoretically derived egalitarian fairness bounds with the empirically achieved egalitarian fairness in fair FL settings.

PDF Details DOI

TMLR Journal 2024 Journal Article

Improve Certified Training with Signal-to-Noise Ratio Loss to Decrease Neuron Variance and Increase Neuron Stability

Tianhao Wei
Ziwei Wang
Peizhi Niu
ABULIKEMU ABUDUWEILI
Weiye Zhao
Casidhe Hutchison
Eric Sample
Changliu Liu

Neural network robustness is a major concern in safety-critical applications. Certified robustness provides a reliable lower bound on worst-case robustness, and certified training methods have been developed to enhance it. However, certified training methods often suffer from over-regularization, leading to lower certified robustness. This work addresses this issue by introducing the concepts of neuron variance and neuron stability, examining their impact on over-regularization and model robustness. To tackle the problem, we extend the Signal-to-Noise Ratio (SNR) into the realm of model robustness, offering a novel perspective and developing SNR-inspired losses aimed at optimizing neuron variance and stability to mitigate over-regularization. Through both empirical and theoretical analysis, our SNR-based approach demonstrates superior performance over existing methods on the MNIST and CIFAR-10 datasets. In addition, our exploration of adversarial training uncovers a beneficial correlation between neuron variance and adversarial robustness, leading to an optimized balance between standard and robust accuracy that outperforms baseline methods.

PDF Details

NeurIPS Conference 2024 Conference Paper

Q-VLM: Post-training Quantization for Large Vision-Language Models

Changyuan Wang
Ziwei Wang
Xiuwei Xu
Yansong Tang
Jie Zhou
Jiwen Lu

In this paper, we propose a post-training quantization framework of large vision-language models (LVLMs) for efficient multi-modal inference. Conventional quantization methods sequentially search the layer-wise rounding functions by minimizing activation discretization errors, which fails to acquire optimal quantization strategy without considering cross-layer dependency. On the contrary, we mine the cross-layer dependency that significantly influences discretization errors of the entire vision-language model, and embed this dependency into optimal quantization strategy searching with low search cost. Specifically, we observe the strong correlation between the activation entropy and the cross-layer dependency concerning output discretization errors. Therefore, we employ the entropy as the proxy to partition blocks optimally, which aims to achieve satisfying trade-offs between discretization errors and the search cost. Moreover, we optimize the visual encoder to disentangle the cross-layer dependency for fine-grained decomposition of search space, so that the search cost is further reduced without harming the quantization accuracy. Experimental results demonstrate that our method compresses the memory by 2. 78x and increase generate speed by 1. 44x about 13B LLaVA model without performance degradation on diverse multi-modal reasoning tasks.

PDF Details DOI

IJCAI Conference 2024 Conference Paper

Real-time Multi-modal Object Detection and Tracking on Edge for Regulatory Compliance Monitoring

Jia Syuen Lim
Ziwei Wang
Jiajun Liu
Abdelwahed Khamis
Reza Arablouei
Robert Barlow
Ryan McAllister

Regulatory compliance auditing in agrifood processing facilities is crucial for upholding the highest standards of quality assurance and traceability. However, the current manual and intermittent approaches to auditing present significant challenges and risks, potentially leading to gaps or loopholes in the system. To address these shortcomings, we introduce a real-time, multi-modal sensing system that utilizes 3D time-of-flight and RGB cameras and leverages unsupervised learning techniques on edge AI devices. The proposed system enables continuous object tracking, leading to improved efficiency in record-keeping and reduced manual labor. We demonstrate the effectiveness of the system in a knife sanitization monitoring scenario, showcasing its capability to overcome occlusion and low-light performance limitations commonly encountered with conventional RGB cameras.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

MCUFormer: Deploying Vision Tranformers on Microcontrollers with Limited Memory

Yinan Liang
Ziwei Wang
Xiuwei Xu
Yansong Tang
Jie Zhou
Jiwen Lu

Due to the high price and heavy energy consumption of GPUs, deploying deep models on IoT devices such as microcontrollers makes significant contributions for ecological AI. Conventional methods successfully enable convolutional neural network inference of high resolution images on microcontrollers, while the framework for vision transformers that achieve the state-of-the-art performance in many vision applications still remains unexplored. In this paper, we propose a hardware-algorithm co-optimizations method called MCUFormer to deploy vision transformers on microcontrollers with extremely limited memory, where we jointly design transformer architecture and construct the inference operator library to fit the memory resource constraint. More specifically, we generalize the one-shot network architecture search (NAS) to discover the optimal architecture with highest task performance given the memory budget from the microcontrollers, where we enlarge the existing search space of vision transformers by considering the low-rank decomposition dimensions and patch resolution for memory reduction. For the construction of the inference operator library of vision transformers, we schedule the memory buffer during inference through operator integration, patch embedding decomposition, and token overwriting, allowing the memory buffer to be fully utilized to adapt to the forward pass of the vision transformer. Experimental results demonstrate that our MCUFormer achieves 73. 62\% top-1 accuracy on ImageNet for image classification with 320KB memory on STM32F746 microcontroller. Code is available at https: //github. com/liangyn22/MCUFormer.

PDF Details

IJCAI Conference 2020 Conference Paper

Human Consensus-Oriented Image Captioning

Ziwei Wang
Zi Huang
Yadan Luo

Image captioning aims to describe an image with a concise, accurate, and interesting sentence. To build such an automatic neural captioner, the traditional models align the generated words with a number of human-annotated sentences to mimic human-like captions. However, the crowd-sourced annotations inevitably come with data quality issues such as grammatical errors, wrong identification of visual objects and sub-optimal sentence focus. During the model training, existing methods treat all the annotations equally regardless of the data quality. In this work, we explicitly engage human consensus to measure the quality of ground truth captions in advance, and directly encourage the model to learn high quality captions with high priority. Therefore, the proposed consensus-oriented method can accelerate the training process and achieve superior performance with only supervised objective without time-consuming reinforcement learning. The novel consensus loss can be implemented into most of the existing state-of-the-art methods, boosting the BLEU-4 performance by maximum relative 12. 47% comparing to the conventional cross-entropy loss. Extensive experiments are conducted on MS-COCO Image Captioning dataset demonstrating the proposed human consensus-oriented training method can significantly improve the training efficiency and model effectiveness.

PDF Details DOI

AAAI Conference 2020 Conference Paper

Learning from the Past: Continual Meta-Learning with Bayesian Graph Neural Networks

Yadan Luo
Zi Huang
Zheng Zhang
Ziwei Wang
Mahsa Baktashmotlagh
Yang Yang

Meta-learning for few-shot learning allows a machine to leverage previously acquired knowledge as a prior, thus improving the performance on novel tasks with only small amounts of data. However, most mainstream models suffer from catastrophic forgetting and insufﬁcient robustness issues, thereby failing to fully retain or exploit long-term knowledge while being prone to cause severe error accumulation. In this paper, we propose a novel Continual Meta- Learning approach with Bayesian Graph Neural Networks (CML-BGNN) that mathematically formulates meta-learning as continual learning of a sequence of tasks. With each task forming as a graph, the intra- and inter-task correlations can be well preserved via message-passing and history transition. To remedy topological uncertainty from graph initialization, we utilize Bayes by Backprop strategy that approximates the posterior distribution of task-speciﬁc parameters with amortized inference networks, which are seamlessly integrated into the end-to-end edge learning. Extensive experiments conducted on the miniImageNet and tieredImageNet datasets demonstrate the effectiveness and efﬁciency of the proposed method, improving the performance by 42. 8% compared with state-of-the-art on the miniImageNet 5-way 1-shot classiﬁcation task.

PDF Details

IJCAI Conference 2016 Conference Paper

Taking Up the Gaokao Challenge: An Information Retrieval Approach

Gong Cheng
Weixi Zhu
Ziwei Wang
Jianghui Chen
Yuzhong Qu

Answering questions in a university's entrance examination like Gaokao in China challenges AI technology. As a preliminary attempt to take up this challenge, we focus on multiple-choice questions in Gaokao, and propose a three-stage approach that exploits and extends information retrieval techniques. Taking Wikipedia as the source of knowledge, our approach obtains knowledge relevant to a question by retrieving pages from Wikipedia via string matching and context-based disambiguation, and then ranks and filters pages using multiple strategies to draw critical evidence, based on which the truth of each option is assessed via relevance-based entailment. It achieves encouraging results on real-life questions in recent history tests, significantly outperforming baseline approaches.

PDF Details