Systematic Analysis of Effectiveness of Artificial Data in Direct Speech-to-Speech Translation

Direct speech-to-speech translation (DS2ST) systems enable the direct translation of spoken language into a target language without relying on text. However, training these systems typically requires parallel speech data, which is often limited in availability. To mitigate this, data augmentation and generation techniques are employed, though these methods only partially enhance performance. To evaluate the effectiveness of generated data, this study conducts a systematic comparative analysis using both real and artificial data in a DS2ST system. We first trained the system using real data, then with an equivalent amount of artificial data, and finally with a combination of both. Our findings indicate that incorporating artificial data resulted in a modest performance improvement of 2.82 BLEU. However, when the system is trained exclusively on artificial data, its performance is 6.12 BLEU lower compared to training with real data alone.