K-ON: Stacking Knowledge on the Head Layer of Large Language Model

Recent advancements in large language models (LLMs) have significantly improved various natural language processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP tasks. However, in knowledge graph (KG) scenarios, entities are the fundamental units and identifying an entity requires at least several tokens. This leads to a granularity mismatch between KGs and natural languages. To address this issue, we propose K-ON, which integrates KG knowledge into the LLM by employing multiple head layers for next k-step prediction. K-ON can not only generate entity-level results in one step, but also enables contrastive loss against entities, which is the most powerful tool in KG representation learning. Experimental results show that K-ON outperforms state-of-the-art methods that incorporate text and even the other modalities.

Authors

• Lingbing Guo College of Computer Science and Technology, Zhejiang University ZJU-Ant Group Joint Lab of Knowledge Graph
• Yichi Zhang College of Computer Science and Technology, Zhejiang University ZJU-Ant Group Joint Lab of Knowledge Graph
• Zhongpu Bo Ant Group
• Zhuo Chen ByteDance Inc., USA
• Mengshu Sun Ant Group
• Zhiqiang Zhang
• Wen Zhang
• Huajun Chen College of Computer Science and Technology, Zhejiang University ZJU-Ant Group Joint Lab of Knowledge Graph Zhejiang Key Laboratory of Big Data Intelligent Computing

Keywords

No keywords are indexed for this paper.