Fangkai Yang

TMLR Journal 2026 Journal Article

VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model

• Mengzhuo Chen
• Jiani zheng
• Lu Wang
• Fangkai Yang
• Chaoyun Zhang
• Lingrui Mei
• Wenjie Yin
• Qingwei Lin

Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI) agents via Reinforcement Learning (RL) faces critical challenges: environment-based RL requires costly interactions, while environment-free methods struggle with distribution shift and reward generalization. We propose an environment-free RL framework that decouples action utility learning from policy optimization by leveraging a pretrained Value Environment Model (VEM), which requires no live environment interaction during policy optimization. VEM predicts value-aligned action utilities directly from offline data, distilling human-like priors about GUI interaction outcomes without requiring next-state prediction or environmental feedback. This avoids compounding errors and enhances resilience to UI changes by focusing on semantic reasoning (e.g., “Does this action advance the user’s goal?”). The framework operates in two stages: (1) pretraining VEM to learn action-level utility signals and (2) guiding policy exploration with frozen VEM signals, enabling layout-agnostic GUI automation. Evaluated across diverse benchmarks including Android-in-the-Wild for mobile apps and Multimodal-Mind2Web for web environments, VEM achieves state-of-the-art or highly competitive performance in both offline and online settings. It significantly outperforms environment-free baselines and matches or exceeds environment-based approaches, crucially without incurring interaction costs. Importantly, VEM demonstrates that robust, generalizable GUI agents can be trained efficiently using semantic-aware action utility prediction, proving effective across distinct interaction platforms like mobile and web. The code is available at https://github.com/microsoft/GUI-Agent-RL.

TMLR Journal 2025 Journal Article

Large Action Models: From Inception to Implementation

• Lu Wang
• Fangkai Yang
• Chaoyun Zhang
• Junting Lu
• Jiaxu Qian
• Shilin He
• Pu Zhao
• Bo Qiao

As AI continues to advance, there is a growing demand for systems that go beyond language-based assistance and move toward intelligent agents capable of performing real-world actions. This evolution requires the transition from traditional Large Language Models (LLMs), which excel at generating textual responses, to Large Action Models (LAMs), designed for action generation and execution within dynamic environments. Enabled by agent systems, LAMs hold the potential to transform AI from passive language understanding to active task completion, marking a significant milestone in the progression toward artificial general intelligence. In this paper, we present a comprehensive framework for developing LAMs, offering a systematic approach to their creation, from inception to deployment. We begin with an overview of LAMs, highlighting their unique characteristics and delineating their differences from LLMs. Using a Windows OS-based agent as a case study, we provide a detailed, step-by-step guide on the key stages of LAM development, including data collection, model training, environment integration, grounding, and evaluation. This generalizable workflow can serve as a blueprint for creating functional LAMs in various application domains. We conclude by identifying the current limitations of LAMs and discussing directions for future research and industrial deployment, emphasizing the challenges and opportunities that lie ahead in realizing the full potential of LAMs in real-world applications.

ICLR Conference 2025 Conference Paper

Self-Evolved Reward Learning for LLMS

• Chenghua Huang
• Zhizhen Fan
• Lu Wang 0029
• Fangkai Yang
• Pu Zhao 0004
• Zeqi Lin
• Qingwei Lin
• Dongmei Zhang 0001

Reinforcement Learning from Human Feedback (RLHF) is a crucial technique for aligning language models with human preferences and is a key factor in the success of modern conversational models like GPT-4, ChatGPT, and Llama 2. A significant challenge in employing RLHF lies in training a reliable RM, which relies on high-quality labels. Typically, these labels are provided by human experts or a stronger AI, both of which can be costly and introduce bias that may affect the language model's responses. As models improve, human input may become less effective in enhancing their performance. This paper explores the potential of using the RM itself to generate additional training data for a more robust RM. Our experiments demonstrate that reinforcement learning from self-feedback outperforms baseline approaches. We conducted extensive experiments with our approach on multiple datasets, such as HH-RLHF and UltraFeedback, and models including Mistral and Llama 3, comparing it against various baselines. Our results indicate that, even with a limited amount of human-labeled data, learning from self-feedback can robustly enhance the performance of the RM, thereby improving the capabilities of large language models.

ECAI Conference 2024 Conference Paper

Nissist: An Incident Mitigation Copilot based on Troubleshooting Guides

• Kaikai An
• Fangkai Yang
• Junting Lu
• Liqun Li
• Zhixing Ren
• Hao Huang
• Lu Wang 0029
• Pu Zhao 0004

Effective incident management is pivotal for the smooth operation of Microsoft cloud services. In order to expedite incident mitigation, service teams gather troubleshooting knowledge into Troubleshooting Guides (TSGs) accessible to On-Call Engineers (OCEs). While automated pipelines are enabled to resolve the most frequent and easy incidents, there still exist complex incidents that require OCEs’ intervention. In addition, TSGs are often unstructured and incomplete, which requires manual interpretation by OCEs, leading to on-call fatigue and decreased productivity, especially among new-hire OCEs. In this work, we propose Nissist which leverages unstructured TSGs and incident mitigation history to provide proactive incident mitigation suggestions, reducing human intervention. Leveraging Large Language Models (LLM), Nissist extracts knowledge from unstructured TSGs and incident mitigation history, forming a comprehensive knowledge base. Its multi-agent system design enhances proficiency in precisely discerning OCE intents, retrieving relevant information, and delivering systematic plans consecutively. Through our user experiments, we demonstrate that Nissist significantly reduce Time to Mitigate (TTM) in incident mitigation, alleviating operational burdens on OCEs and improving service reliability. Our webpage is available at https: //aka. ms/nissist.

IJCAI Conference 2023 Conference Paper

Measuring Acoustics with Collaborative Multiple Agents

• Yinfeng Yu
• Changan Chen
• Lele Cao
• Fangkai Yang
• Fuchun Sun

As humans, we hear sound every second of our life. The sound we hear is often affected by the acoustics of the environment surrounding us. For example, a spacious hall leads to more reverberation. Room Impulse Responses (RIR) are commonly used to characterize environment acoustics as a function of the scene geometry, materials, and source/receiver locations. Traditionally, RIRs are measured by setting up a loudspeaker and microphone in the environment for all source/receiver locations, which is time-consuming and inefficient. We propose to let two robots measure the environment's acoustics by actively moving and emitting/receiving sweep signals. We also devise a collaborative multi-agent policy where these two robots are trained to explore the environment's acoustics while being rewarded for wide exploration and accurate prediction. We show that the robots learn to collaborate and move to explore environment acoustics while minimizing the prediction error. To the best of our knowledge, we present the very first problem formulation and solution to the task of collaborative environment acoustics measurements with multiple agents.

ECAI Conference 2020 Conference Paper

Group Behavior Recognition Using Attention- and Graph-Based Neural Networks

• Fangkai Yang
• Wenjie Yin
• Tetsunari Inamura
• Mårten Björkman
• Christopher Peters 0001

When a conversational group is approached by a newcomer who wishes to join it, the group may dynamically react by adjusting their positions and orientations in order to accommodate it. These reactions represent important cues to the newcomer about if and how they should plan their approach. The recognition and analysis of such socially complaint dynamic group behaviors have rarely been studied in depth and remain a challenging problem in social multi-agent systems. In this paper, we present novel group behavior recognition models, attention-based and graph-based, that consider behaviors on both the individual and group levels. The attention-based category consists of Approach Group Net (AGNet) and Approach Group Transformer (AGTransformer). They share a similar architecture and use attention mechanisms to encode both temporal and spatial information on both the individual and group levels. The graph-based models consist of Approach Group Graph Convolutional Networks (AG-GCN), which combine Multi-Spatial-Temporal Graph Convolutional Networks (MST-GCN) on the individual level and Graph Convolutional Networks (GCN) on the group level, with multi-temporal stages. The individual level learns the spatial and temporal movement patterns of each agent, while the group level captures the relations and interactions of multiple agents. In order to train and evaluate these models, we collected a full-body motion-captured dataset of multiple individuals in conversational groups. Experiments performed using our models to recognize group behaviors from the collected dataset show that AG-GCN, with additional distance and orientation information, achieves the best performance. We also present a multi-agent interaction use case in a virtual environment to show how the models can be practically applied.

AAMAS Conference 2019 Conference Paper

Interpretable Automated Machine Learning in Maana™ Knowledge Platform

• Alexander Elkholy
• Fangkai Yang
• Steven Gustafson

AAAI Conference 2019 Short Paper

Logic-Based Sequential Decision-Making

• Daoming Lyu
• Fangkai Yang
• Bo Liu
• Daesub Yoon

Deep reinforcement learning (DRL) has gained great success by learning directly from high-dimensional sensory inputs, yet is notorious for the lack of interpretability. Interpretability of the subtasks is critical in hierarchical decision-making as it increases the transparency of black-box-style DRL approach and helps the RL practitioners to understand the high-level behavior of the system better. In this paper, we introduce symbolic planning into DRL and propose a framework of Symbolic Deep Reinforcement Learning (SDRL) that can handle both high-dimensional sensory inputs and symbolic planning. The task-level interpretability is enabled by relating symbolic actions to options. This framework features a planner – controller – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively. The three components cross-fertilize each other and eventually converge to an optimal symbolic plan along with the learned subtasks, bringing together the advantages of long-term planning capability with symbolic knowledge and end-to-end reinforcement learning directly from a high-dimensional sensory input. Experimental results validate the interpretability of subtasks, along with improved data efficiency compared with state-of-the-art approaches.

AAMAS Conference 2019 Conference Paper

Priority driven Local Optimization for Crowd Simulation

• Himangshu Saikia
• Fangkai Yang
• Christopher Peters

We provide an initial model and preliminary findings of a lookahead based local optimization scheme for collision resolution between agents in large goal-directed crowd simulations. Considering crowd simulation to be a global optimization problem, we break down this large problem into smaller problems where each potential collision resolution step is independently optimized in terms of a criticality measure. Agents resolved earlier in order of criticality, maintain the optimized velocity obtained, for the resolution of agents that come later in that order. Hence, the problem is converted to a low dimensional optimization problem of one or two agents where all other obstacles are static or deterministically dynamic. We illustrate the performance of our method on four well known test scenarios.

AAAI Conference 2019 Conference Paper

SDRL: Interpretable and Data-Efficient Deep Reinforcement Learning Leveraging Symbolic Planning

• Daoming Lyu
• Fangkai Yang
• Bo Liu
• Steven Gustafson

Deep reinforcement learning (DRL) has gained great success by learning directly from high-dimensional sensory inputs, yet is notorious for the lack of interpretability. Interpretability of the subtasks is critical in hierarchical decision-making as it increases the transparency of black-box-style DRL approach and helps the RL practitioners to understand the high-level behavior of the system better. In this paper, we introduce symbolic planning into DRL and propose a framework of Symbolic Deep Reinforcement Learning (SDRL) that can handle both high-dimensional sensory inputs and symbolic planning. The task-level interpretability is enabled by relating symbolic actions to options. This framework features a planner – controller – meta-controller architecture, which takes charge of subtask scheduling, data-driven subtask learning, and subtask evaluation, respectively. The three components cross-fertilize each other and eventually converge to an optimal symbolic plan along with the learned subtasks, bringing together the advantages of long-term planning capability with symbolic knowledge and end-to-end reinforcement learning directly from a high-dimensional sensory input. Experimental results validate the interpretability of subtasks, along with improved data efficiency compared with state-of-the-art approaches.

IROS Conference 2019 Conference Paper

Task-Motion Planning with Reinforcement Learning for Adaptable Mobile Service Robots

• Yuqian Jiang
• Fangkai Yang
• Shiqi Zhang 0001
• Peter Stone 0001

Task-motion planning (TMP) addresses the problem of efficiently generating executable and low-cost task plans in a discrete space such that the (initially unknown) action costs are determined by motion plans in a corresponding continuous space. A task-motion plan for a mobile service robot that behaves in a highly dynamic domain can be sensitive to domain uncertainty and changes, leading to suboptimal behaviors or execution failures. In this paper, we propose a novel framework, TMP-RL, which is an integration of TMP and reinforcement learning (RL), to solve the problem of robust TMP in dynamic and uncertain domains. The robot first generates a low-cost, feasible task-motion plan by iteratively planning in the discrete space and updating relevant action costs evaluated by the motion planner in continuous space. During execution, the robot learns via model-free RL to further improve its task-motion plans. RL enables adaptability to the current domain, but can be costly with regards to experience; using TMP, which does not rely on experience, can jump-start the learning process before executing in the real world. TMP-RL is evaluated in a mobile service robot domain where the robot navigates in an office area, showing significantly improved adaptability to unseen domain dynamics over TMP and task planning (TP)-RL methods.

AAMAS Conference 2018 Conference Paper

Investigating Social Distances between Humans, Virtual Humans and Virtual Robots in Mixed Reality

• Christopher Peters
• Chengjie Li
• Fangkai Yang
• Vanya Avramova
• Gabriel Skantze

Mixed reality environments offer new potentials for the design of compelling social interaction experiences with virtual characters. In this paper, we summarise initial experiments we are conducting in which we measure comfortable social distances between humans, virtual humans and virtual robots in mixed reality environments. We consider a scenario in which participants walk within a comfortable distance of a virtual character that has its appearance varied between a male and female human, and a standard- and humanheight virtual Pepper robot. Our studies in mixed reality thus far indicate that humans adopt social zones with artificial agents that are similar in manner to human-human social interactions and interactions in virtual reality.

IJCAI Conference 2018 Conference Paper

PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Making

• Fangkai Yang
• Daoming Lyu
• Bo Liu
• Steven Gustafson

Reinforcement learning and symbolic planning have both been used to build intelligent autonomous agents. Reinforcement learning relies on learning from interactions with real world, which often requires an unfeasibly large amount of experience. Symbolic planning relies on manually crafted symbolic knowledge, which may not be robust to domain uncertainties and changes. In this paper we present a unified framework PEORL that integrates symbolic planning with hierarchical reinforcement learning (HRL) to cope with decision-making in dynamic environment with uncertainties. Symbolic plans are used to guide the agent's task execution and learning, and the learned experience is fed back to symbolic knowledge to improve planning. This method leads to rapid policy search and robust symbolic plans in complex domains. The framework is tested on benchmark domains of HRL.

IJCAI Conference 2016 Conference Paper

Planning with Task-Oriented Knowledge Acquisition for a Service Robot

• Kai Chen
• Fangkai Yang
• Xiaoping Chen

We propose a framework for a service robot to behave intelligently in domains that contain incomplete information, underspecified goals and dynamic change. Human robot interaction (HRI), sensing actions and physical actions are uniformly formalized in action language BC. An answer set solver is called to generate plans that guide the robot to acquire task-oriented knowledge and execute actions to achieve its goal, including interacting with human to gather information and sensing the environment to help motion planning. By continuously interpreting and grounding useful sensing information, robot is able to use contingent knowledge to adapt to unexpected changes and faults. We evaluate the approach on service robot KeJia that serves drink to guests, a testing benchmark for general-purpose service robot proposed by RoboCup@Home competition.

ICAPS Conference 2014 Conference Paper

Planning in Action Language BC while Learning Action Costs for Mobile Robots

• Piyush Khandelwal
• Fangkai Yang
• Matteo Leonetti
• Vladimir Lifschitz
• Peter Stone 0001

The action language BC provides an elegant way of formalizing dynamic domains which involve indirect effects of actions and recursively defined fluents. In complex robot task planning domains, it may be necessary for robots to plan with incomplete information, and reason about indirect or recursive action effects. In this paper, we demonstrate how BC can be used for robot task planning to solve these issues. Additionally, action costs are incorporated with planning to produce optimal plans, and we estimate these costs from experience making planning adaptive. This paper presents the first application of BC on a real robot in a realistic domain, which involves human-robot interaction for knowledge acquisition, optimal plan generation to minimize navigation time, and learning for adaptive planning.

KR Conference 2014 Conference Paper

The Semantics of Gringo and Infinitary Proposition Formulas

• Amelia Harrison
• Vladimir Lifschitz
• Fangkai Yang

IJCAI Conference 2013 Conference Paper

Action Language BC: Preliminary Report

• Joohyung Lee
• Vladimir Lifschitz
• Fangkai Yang

The action description languages B and C have significant common core. Nevertheless, some expressive possibilities of B are difficult or impossible to simulate in C, and the other way around. The main advantage of B is that it allows the user to give Prolog-style recursive definitions, which is important in applications. On the other hand, B solves the frame problem by incorporating the commonsense law of inertia in its semantics, which makes it difficult to talk about fluents whose behavior is described by defaults other than inertia. In C and in its extension C+, the inertia assumption is expressed by axioms that the user is free to include or not to include, and other defaults can be postulated as well. This paper defines a new action description language, called BC, that combines the attractive features of B and C+. Examples of formalizing commonsense domains discussed in the paper illustrate the expressive capabilities of BC and the use of answer set solvers for the automation of reasoning about actions described in this language.

JELIA Conference 2010 Conference Paper

Translating First-Order Causal Theories into Answer Set Programming

• Vladimir Lifschitz
• Fangkai Yang

Abstract Nonmonotonic causal logic became a basis for the semantics of several expressive action languages. Norman McCain and Paolo Ferraris showed how to embed propositional causal theories into logic programming, and this work paved the way to the use of answer set solvers for answering queries about actions described in causal logic. In this paper we generalize these embeddings to first-order causal logic—a system that has been used to simplify the semantics of variables in action descriptions.