Large Action Models: From Inception to Implementation

As AI continues to advance, there is a growing demand for systems that go beyond language-based assistance and move toward intelligent agents capable of performing real-world actions. This evolution requires the transition from traditional Large Language Models (LLMs), which excel at generating textual responses, to Large Action Models (LAMs), designed for action generation and execution within dynamic environments. Enabled by agent systems, LAMs hold the potential to transform AI from passive language understanding to active task completion, marking a significant milestone in the progression toward artificial general intelligence. In this paper, we present a comprehensive framework for developing LAMs, offering a systematic approach to their creation, from inception to deployment. We begin with an overview of LAMs, highlighting their unique characteristics and delineating their differences from LLMs. Using a Windows OS-based agent as a case study, we provide a detailed, step-by-step guide on the key stages of LAM development, including data collection, model training, environment integration, grounding, and evaluation. This generalizable workflow can serve as a blueprint for creating functional LAMs in various application domains. We conclude by identifying the current limitations of LAMs and discussing directions for future research and industrial deployment, emphasizing the challenges and opportunities that lie ahead in realizing the full potential of LAMs in real-world applications.

Authors

• Lu Wang
• Fangkai Yang
• Chaoyun Zhang
• Junting Lu Peking University
• Jiaxu Qian
• Shilin He
• Pu Zhao Northeastern University
• Bo Qiao
• He Huang School of Computer Science and Technology, Soochow University
• Si Qin
• Qisheng Su
• Jiayi Ye
• Yudi Zhang Eindhoven University of Technology
• Jian-Guang Lou Microsoft Research Asia
• Qingwei Lin Microsoft Research
• Saravan Rajmohan Microsoft
• Dongmei Zhang Peking University
• Qi Zhang School of Computer Science, Fudan University, Shanghai, China Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China

Keywords

No keywords are indexed for this paper.