Context Adaptive Memory-Efficient LLM Inference for Edge Multi-Agent Systems

Large Language Models (LLMs) excel at multi-document QA, summarization, code generation, and other language-intensive tasks, yet they demand substantial memory resources for storing keyvalue (KV) caches and processing attention in long-context scenarios. These requirements often prohibit on-device or edge deployments in multi-agent systems (MAS), where multiple agents share or update contextual information and need efficient inference pipelines. We present CASK (Context-Adaptive Sparse Key-value), an inference-time strategy that reduces memory usage while preserving strong performance on extended contexts. CASK addresses this challenge with two complementary mechanisms: a dynamic sparse attention module—a lightweight, meta-learned component—that identifies the most relevant context tokens, and an adaptive KV-cache compression technique that dynamically quantizes and prunes less critical key-value pairs based on usage frequency and recency. These innovations enable near-lossless performance on long-context tasks while cutting memory usage by up to 40% and boosting inference speed by as much as 20%. Evaluations on LongBench [2] and multi-agent benchmarks show that CASK maintains over 95% of baseline accuracy while allowing more agents or extended histories under tight GPU budgets. Integration into a vision-language agent for collaborative, multimodal contexts underscores its practicality for resource-constrained LLM deployments in MAS.

Authors

• Hamza Mohammed Samsung Research America
• Hang Yin University of Copenhagen
• Sai Chand Boyapati Samsung Research America

Keywords

• Multi-Agent Systems
• Natural Language Generation
• Dimensionality Reduction And Manifold Learning
• Heuristic Function Construction
• Caching And Paging Algorithms
• Multi-Agent Learning
• Computer Vision
• Reinforcement Learning
• Multi-Agent Reinforcement
• Learning
• Active Learning
• Neural Networks
• Supervised Learning
• Online Learning Algorithms