Research on Big Data Processing and Analysis Based on Spark SQL

Based on the big data processing engine Spark SQL technology, this paper studies and analyzes the agricultural recommendation data set in the Kaggle data science and machine learning competition platform. The data set mainly includes N, P, K values, PH values, temperature, humidity and rainfall in the soil of 22 crops such as rice, banana, apple and coffee. In this paper, the analysis of pH extremes, the correlation between temperature and humidity, the influence of rainfall on crop types, the distribution of crop types under specific conditions, rainfall, pH and other characteristics in different crop types are analyzed. Build interactive charts with Pandas and Pyecharts to visualize correlations between data fields.