Learning to Surface Deep Web Content

We propose a novel deep web crawling framework based on reinforcement learning. The crawler is regarded as an agent and deep web database as the environment. The agent perceives its current state and submits a selected action (query) to the environment according to Q-value. Based on the framework we develop an adaptive crawling method. Experimental results show that it outperforms the state of art methods in crawling capability and breaks through the assumption of full-text search implied by existing methods.

Authors

• Zhaohui Wu
• Lu Jiang Dalian Martime University
• Qinghua Zheng School of Computer Science and Technology, Xi'an Jiaotong University, China Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi'an Jiaotong University, China
• Jun Liu Carnegie Mellon University

Keywords

No keywords are indexed for this paper.