Isolation Forest Based Anomaly Detection Framework on Non-IID Data

Anomaly detection is a significant but challenging data mining task in a wide range of applications. Different domains usually use different ways to measure the characteristics of data and to define the anomaly types. As a result, it is a big challenge to develop a versatile anomaly detection framework that can be universally applied with satisfactory performance in most, if not all, applications. In this article, we propose a generic isolation forest based ensemble framework named EDBHiForest, which can be universally applied to data spaces with arbitrary distance measures. It is realized through embedding the isolation forest structure with extended distance-based hashing (EDBH), which can significantly enhance the versatility and applicability of isolation forest based anomaly detection. This framework overcomes the limitations of existing isolation forest based methods that can only be applied to datasets with a very limited range of distance measure types. Extensive experiments on various non-independent and identically distributed datasets demonstrate the effectiveness and efficiency of our approach.

Authors

• Haolong Xiang Nanjing University of Information Science and Technology
• Jiayu Wang Shanghai Key Lab of Intell. Info. Processing, School of Computer Science, Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing
• Kotagiri Ramamohanarao
• Zoran Salcic
• Wanchun Dou Nanjing University, China
• Xuyun Zhang Macquarie University

Keywords

• Anomaly detection
• Data mining
• Measurement
• Extraterrestrial measurements
• Hash functions
• Intelligent systems
• Isolation Forest
• non-IID Data
• Anomaly Detection Framework
• Big Data
• Distancing Measures
• Similarity Measure
• General Framework
• Extensive Experiments
• Forest Structure
• Smart Manufacturing
• Dynamic Time Warping
• Types Of Anomalies
• Arbitrary Space
• Arbitrary Measure
• Anomaly Detection Methods
• Data Mining Tasks
• Locality Sensitive Hashing
• Window Size
• Time Complexity
• Data Instances
• Different Types Of Datasets
• Hash Function
• Types Of Datasets
• Anomaly Score
• Pre-training Stage
• Ensemble Method
• Area Under Curve
• Similar Complexity
• Tree Height