Preference-driven Knowledge Distillation for Few-shot Node Classification

Graph neural networks (GNNs) can efficiently process text-attributed graphs (TAGs) due to their message-passing mechanisms, but their training heavily relies on the human-annotated labels. Moreover, the complex and diverse local topologies of nodes of real-world TAGs make it challenging for a single mechanism to handle. Large language models (LLMs) perform well in zero-/few-shot learning on TAGs but suffer from a scalability challenge. Therefore, we propose a preference-driven knowledge distillation (PKD) framework to synergize the complementary strengths of LLMs and various GNNs for few-shot node classification. Specifically, we develop a GNN-preference-driven node selector that effectively promotes prediction distillation from LLMs to teacher GNNs. To further tackle nodes' intricate local topologies, we develop a node-preference-driven GNN selector that identifies the most suitable teacher GNN for each node, thereby facilitating tailored knowledge distillation from teacher GNNs to the student GNN. Extensive experiments validate the efficacy of our proposed framework in few-shot node classification on real-world TAGs. Our code can be available at.

Authors

• Xing Wei Xi'an Jiaotong University
• Chunchun Chen
• Rui Fan Key Laboratory of Analog Integrated Circuits and Systems (Ministry of Education) School of Integrated Circuits, Xidian University, Xi’an 710071, China
• Xiaofeng Cao Jilin University
• Sourav Medya UIC
• Wei Ye Tongji University

Keywords

No keywords are indexed for this paper.