Frequent Itemset Mining in Big Data with Efficient Distributed Single Scan Algorithm Based on Spark

Frequent itemset mining is a data mining technique used to discover frequent patterns in datasets, beneficial for prediction, association rule mining, classification, etc.The Single Scan frequent itemsets mining (SSFIM) is a recent iterative algorithm requires a single scan of the transactional database to extract the frequent itemsets.It has a unique feature to allow the generation of a fixed number of candidate itemsets, independently from the minimum support threshold.While suitable for small data, it becomes impractical for big data.To address this, the MapReduce framework enables running Single Scan on large transactional data, but its iterative limitations hinder performance.In this paper, we propose an enhanced version of SSFIM for Big Data, using distributive computation, called Distributed Single Scan on Spark (DSSS).It has a feature to allow the generation of a fixed number of candidate itemsets, independently from the minimum support threshold, which intuitively allows reducing the cost in terms of runtime for large databases.Our algorithm is implemented on the Spark framework, which incorporates resilient distributed datasets and performs in-memory processing, optimizing execution time, and it has small network communication because we shared a hash table with all the executors of cluster by using a special feature of Spark framework called broadcast variable.Experience has shown that the DSSS algorithm is efficient, scalable and demonstrably outperforms by a significant margin three advanced state-of-the-art methods MRSSFIM, DAPRIORI and the well-known algorithm PFP on large databases.