Author name cluster

Junyang Wang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

4 papers

1 author row

NeurIPS Conference 2025 Conference Paper

Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation

Yuyang Wanyan
Xi Zhang
Haiyang Xu
Haowei Liu
Junyang Wang
Jiabo Ye
Yutong Kou
Ming Yan

In recent years, Multimodal Large Language Models (MLLMs) have been extensively utilized for multimodal reasoning tasks, including Graphical User Interface (GUI) automation. Unlike general offline multimodal tasks, GUI automation is executed in online interactive environments, necessitating step-by-step decision-making based on the real-time status of the environment. This task has a lower tolerance for decision-making errors at each step, as any mistakes may cumulatively disrupt the process and potentially lead to irreversible outcomes like deletions or payments. To address these issues, we introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution, by reasoning about the potential outcome and correctness of actions. Specifically, we propose a Suggestion-aware Group Relative Policy Optimization (S-GRPO) strategy to construct our pre-operative critic model GUI-Critic-R1, incorporating a novel suggestion reward to enhance the reliability of the model's feedback. Furthermore, we develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic data. Static experiments on the GUI-Critic-Test across both mobile and web domains reveal that our GUI-Critic-R1 offers significant advantages in critic accuracy compared to current MLLMs. Dynamic evaluation on GUI automation benchmark further highlights the effectiveness and superiority of our model, as evidenced by improved success rates and operational efficiency. The code is available at https: //github. com/X-PLUG/MobileAgent/tree/main/GUI-Critic-R1.

PDF Details

NeurIPS Conference 2024 Conference Paper

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Junyang Wang
Haiyang Xu
Haitao Jia
Xi Zhang
Ming Yan
Weizhou Shen
Ji Zhang
Fei Huang

Mobile device operation tasks are increasingly becoming a popular multi-modal AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MLLM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks — task progress navigation and focus content navigation — are difficult to effectively solve under the single-agent architecture of existing work. This is due to the overly long token sequences and the interleaved text-image data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent condenses lengthy, interleaved image-text history operations and screens summaries into a pure-text task progress, which is then passed on to the decision agent. This reduction in context length makes it easier for decision agent to navigate the task progress. To retain focus content, we design a memory unit that updates with task progress by decision agent. Additionally, to correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistake accordingly. Experimental results indicate that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture of Mobile-Agent. The code is open-sourced at https: //github. com/X-PLUG/MobileAgent.

PDF Details DOI

IJCAI Conference 2023 Conference Paper

From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping

Junyang Wang
Ming Yan
Yi Zhang
Jitao Sang

With the development of Vision-Language Pre-training Models (VLPMs) represented by CLIP and ALIGN, significant breakthroughs have been achieved for association-based visual tasks such as image classification and image-text retrieval by the zero-shot capability of CLIP without fine-tuning. However, CLIP is hard to apply to generation-based tasks. This is due to the lack of decoder architecture and pre-training tasks for generation. Although previous works have created generation capacity for CLIP through additional language models, a modality gap between the CLIP representations of different modalities and the inability of CLIP to model the offset of this gap, which results in the failure of the concept to transfer across modes. To solve the problem, we try to map images/videos to the language modality and generate captions from the language modality. In this paper, we propose the K-nearest-neighbor Cross-modality Mapping (Knight), a zero-shot method from association to generation. With vision-free unsupervised training, Knight achieves state-of-the-art performance in zero-shot methods for image captioning and video captioning.

PDF Details DOI

YNICL Journal 2019 Journal Article

Structural connectome alterations in patients with disorders of consciousness revealed by 7-tesla magnetic resonance imaging

Xufei Tan
Zhen Zhou
Jian Gao
Fanxia Meng
Yamei Yu
Jie Zhang
Fangping He
Ruili Wei

Although the functional connectivity of patients with disorders of consciousness (DOC) has been widely examined, less is known about brain white matter connectivity. The aim of this study was to explore structural network alterations for the diagnosis and prognosis of patients with chronic DOC. Eleven DOC patients and 11 sex- and age-matched controls were included in the study. Participants underwent diffusion magnetic resonance imaging (MRI) and T1-weighted structural MRI at 7 tesla (7 T). Graph-theoretical analysis and network-based statistics were used to analyze the group differences. Two patients were scanned twice for a longitudinal study to examine the relationship between connectome metrics and the patients' prognoses. Compared with healthy controls, DOC patients showed significantly elevated transitivity (p < .001), local efficiency (p = .009), and clustering coefficient (p = .039). When comparing the connectome metrics within the three groups (healthy controls, minimally conscious state (MCS), and vegetative state/unresponsive wakefulness syndrome (VS/UWS)), significant group differences were observed in transitivity (p < .001) and local efficiency (p = .031). Significantly increased transitivity was observed in vegetative state/unresponsive wakefulness syndrome compared with minimally conscious state (p = .0217, Bonferroni corrected). Transitivity showed significant negative correlations with the Coma Recovery Scale-Revised score (r = -0.6902, p = .023), consistent with the longitudinal study results. A subnetwork with significantly decreased structural connections was identified using network-based statistical analysis comparing DOC patients with healthy controls, which was mainly located in the frontal cortex, limbic system, and occipital and parietal lobes. This preliminary study suggests that graph theoretical approaches for assessing white matter connectivity may enable various states of DOC to be distinguished. Of the metrics analyzed, transitivity had a critical role in distinguishing the diagnostic groups. Larger cohorts will be necessary to confirm the predictive value of 7 T MRI in the prognosis of DOC patients.

Details DOI