Real-time high-throughput cotton phenotyping using distributed computing and deep learning

In this paper, we present an approach for real-time high-throughput cotton phenotyping using distributed computing and deep learning. The objective of this study is to develop a big data pipeline to efficiently ingest and process large amounts of image data collected from cotton fields. Specifically, we employ a cluster of Raspberry Pi's in a primary-replica architecture and use Apache Hadoop, Apache Spark, and tiny-YOLO to process cotton image data from field environment for mapping bloom appearance and frequency throughout a growing season. Our developed cluster is improved with respect to our previous work by transitioning the programming model from Apache Hadoop's MapReduce to Apache Spark for real-time processing. We feed cotton image data collected from a research field in Tifton, GA into our cluster's distributed file system and submit job requests to process the data in parallel. The cotton image data is processed in a distributed fashion using Spark for three imaging tasks: (1) image splitting; (2) bloom detection using our pre-trained tiny-YOLO model and OpenCV; and (3) spatio-temporal map creation. Additionally, we compare our 4-node cluster performance with 1-, 2-, 3-node clusters using both Apache Spark and MapReduce. The results indicate that Spark is at least 5 times faster than MapReduce for processing cotton image data on our cluster. Moreover, this work is the first to develop a real-time, open-source, low-cost, distributed computing pipeline for high-throughput cotton phenotyping in field-based agriculture. • Development of an open-source distributed computing architecture using Apache Hadoop, Apache Spark, and tiny-YOLO for high-throughput cotton plant phenotyping. • Apache Spark provides near real-time processing of cotton image data on a cluster of Raspberry Pi 4's. • Developed cluster is capable of processing cotton image data in parallel.