D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

Developing intelligent agents capable of operating a wide range of Graphical User Interfaces (GUIs) with human-level proficiency is a key milestone on the path toward Artificial General Intelligence. While most existing datasets and benchmarks for training and evaluating GUI agents are static and idealized, failing to reflect the complexity and unpredictability of real-world environments, particularly the presence of anomalies. To bridge this research gap, we propose D-GARA, a dynamic benchmarking framework, to evaluate Android GUI agent robustness in real-world anomalies. D-GARA introduces a diverse set of real-world anomalies that GUI agents commonly face in practice, including interruptions such as permission dialogs, battery warnings, and update prompts. Based on D-GARA framework, we construct and annotate a benchmark featuring commonly used Android applications with embedded anomalies to support broader community research. Comprehensive experiments and results demonstrate substantial performance degradation in state-of-the-art GUI agents when exposed to anomaly-rich environments, highlighting the need for robustness-aware learning. D-GARA is modular and extensible, supporting the seamless integration of new tasks, anomaly types, and interaction scenarios to meet specific evaluation goals.

Authors

• Sen Chen School of Artificial Intelligence, Xidian University, China
• Tong Zhao State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences
• Yi Bin School of Computer Science and Technology, Tongji University
• Fei Ma Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
• Wenqi Shao The Chinese University of Hong Kong
• Zheng Wang National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China

Keywords

No keywords are indexed for this paper.