ScreenAgent: A Vision Language Model-driven Computer Control Agent

Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphical User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing daily computer tasks. Finally, we train a model, ScreenAgent, which achieves comparable computer control capabilities to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code and more detailed information are at https: //github. com/niuzaisheng/ScreenAgent.

Authors

• Runliang Niu School of Artificial Intelligence, Jilin University
• Jindong Li
• Shiqi Wang City University of Hong Kong
• Yali Fu
• Xiyu Hu
• Xueyuan Leng
• He Kong Southern University of Science and Technology
• Yi Chang Jilin University Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Ministry of Education, China
• Qi Wang

Keywords

• Agent-based and Multi-agent Systems: MAS: Human-agent interaction
• Computer Vision: CV: Vision, language and reasoning
• Natural Language Processing: NLP: Dialogue and interactive systems
• Natural Language Processing: NLP: Resources and evaluation