Design and Modeling of a New File Transfer Architecture to Reduce Undetected Errors Evaluated in the FABRIC Testbed

Ensuring data integrity for petabyte-scale file transfers is critical for scientific applications. As packet sizes increase, so does the likelihood of undetected errors. Multi-Level Error Detection (MLED) is a recursive architecture that leverages in-network resources to reduce undetected error probability (UEP) in file transfer. MLED organizes communication in layers at different levels, with each layer implementing configurable policies. Through experimentation on the FABRIC testbed, we show that while the traditional---transport and data link layer---error detection results in corrupt file transfers requiring retransmission, MLED detects and corrects these errors at intermediate levels, when an adversarial error model is used. MLED thus achieves a 100% gain in goodput reaching over 800 Mbps on a single connection with no appreciable increase in delay.