Automated Synthesis of Distributed Code from Sequential Snippets Using Deep Learning

Processing big data poses a significant challenge when transitioning from sequential to distributed code, primarily due to the extensive scale at which data is handled. This complexity arises from both syntax and semantic differences between the codes. Unfortunately, current methods are inefficient and require more effective automated solutions. To address this problem, we utilized the Transformer-based BERT model because of its exceptional capability to understand and capture deep contextual relationships in large datasets. Our method involved creating two comprehensive datasets containing 10k and 100k sequential code snippets paired with their corresponding PySpark API calls. We optimized the BERT model by fine-tuning it to predict distributed API calls for previously unseen sequential snippets. For the 10k dataset, the model demonstrated robustness, achieving a training accuracy of 99.36%, a test accuracy of 99.7%, a Balanced Accuracy of 99.78%, and an F1 Score of 0.997. In contrast, the 100k dataset metrics were equally impressive, with a training accuracy of 89.48%, a test accuracy of 99.99%, a Balanced Accuracy of 99.98%, and an F1 Score of 0.999. Our work demonstrates the feasibility of automating the sequential-to-distributed code transition with notable precision.