Weather Prediction from Historical Data using Big Data Technologies

Weather forecasting is an important function in a variety of activities such as agriculture, environment monitoring, and disaster management among others. Today with the development of technology, vast amounts of meteorological data are able to be collected and analyzed with the intention of improving prediction accuracy. The purpose of this work is to analyze the contribution of Big Data frameworks such as, Hadoop, MapReduce and Apache Spark in effective weather prediction processes. In a distributed system, historical weather data, satellite images and sensor readings are kept and manipulated through the HDFS storage. For effective data manipulation, MapReduce and Spark are used along with Hive and Pig for searching large datasets. Predictive analytics are performed using machine learning algorithms included in the MLlib package of Spark. These results are presented using the Tableau application, which shows the speed and accuracy of this Big Data approach in comparison to classical ones. The proposed work ends with a look into the prospects of Big Data in the field of weather forecasting and relevant problems in this area.