A extensive pipeline model for spatial data processing based on streaming computation

Current mainstream ETL data tools lack support for spatial data processing. Generally, different spatial processing compositions are required to satisfy different application’s requirements. An extensive spatial data streaming processing pipeline model is designed to support its effective and flexible ETL processing. By chaining and assembling different data processing plugins into an uniform spatial data processing pipeline streaming workflow, effective and configurable spatial data transform processing is achieved. Besides, a floating vehicle trajectory data streaming processing prototype system based on proposed model is implemented to verify its practicality. To evaluate proposed model’s efficiency, complete large-scale floating vehicle trajectory data processing workflow which includes extracting, transforming and loading, is conducted with developed prototype system. The result shows that the prototype system processes 10,000, 100,000, 1,000,000 and 11,316,149 data volumes in 2 seconds, 13 seconds, 127 seconds, and 1600 seconds respectively, which proves that the proposed model is available and efficient.