Bridging Scale Discrepancies in Robotic Control via Language-Based Action Representations

Recent end-to-end robotic manipulation research increasingly adopts architectures inspired by large language models to enable robust manipulation. However, a critical challenge arises from severe distribution shifts between robotic action data, primarily due to substantial numerical variations in action commands across diverse robotic platforms and tasks, hindering the effective transfer of pretrained knowledge. To address this limitation, we propose a semantically grounded linguistic representation to normalize actions for efficient pretraining. Unlike conventional discretized action representations that are sensitive to numerical scales, the motion representation specifically disregards numeric scale effects, emphasizing directionality instead. This abstraction mitigates distribution shifts, yielding a more generalizable pretraining representation. Moreover, using the motion representation narrows the feature distance between action tokens and standard vocabulary tokens, mitigating modality gaps. Multi-task experiments on two benchmarks demonstrate that the proposed method significantly improves generalization performance and transferability in robotic manipulation tasks.

Authors

• Yuchi Zhang Harbin Institute of Technology
• Churui Sun Harbin Institute of Technology
• Shiqi Liang Harbin Institute of Technology
• Diyuan Liu iFLYTEK Research
• Chao Ji iFLYTEK Research
• Weinan Zhang Shanghai Jiao Tong University
• Ting Liu Tsinghua University

Keywords

No keywords are indexed for this paper.