Protocol Transformations Across OSI Network Stack Layers for Attack, Evasion, and Defense

Network endpoints frequently contend with errors and deviations within protocols. Many factors account for these deviations including noise, tampering, and algorithm implementations. Intermediate nodes are expected to modify instantiated protocols and not guarantee correctness. This ability to modify traffic enables all sides of network security to alter security and performance properties of protocols, and we define this intermediary modification of an instantiated protocol as a transformation. Protocol transformations traverse layers of the OSI reference model and changes a protocol's time series byte sequence. Within this thesis, we show that this framework applies to multiple domains and protocols. Common examples of transformations changing security properties include firewalls filtering packets or proxies tunneling traffic. Transformations for evasion, defense, and attack face unique challenges. Evasion transformations aim to maintain privacy or avoid detection. Within TARN (traffic analysis resistant network), we show that IP address hopping mutates IP at layers 2-4 and that additional traffic inter-packet timing and packet size normalization reduces traffic fingerprinting effectiveness to approximately random guessing. Application layer detection of parrot and mimicry transformations is another problem for evaders. The problem of mimicry can be avoided by maintaining sessions on the target protocol such as the video game Minecraft. Using Minecraft bots with embedded Hidden Markov Models (HMMs) achieved approximately 350 Kbit/s throughput, and a Minecraft Server operated as an oracle to verify protocol correctness. HMMs can generate indistinguishable action sequences based off the chi-squared test for homogeneity, but embedded HMMs have limitations with tracking application state while using oracles. Attack transformations aim to cause protocol errors or extract data. We consider a stealthy targeted packet dropping attack against the Dedicated Short Range Communication Protocol used as a traffic light replacement. Comparing HMMs to Support Vector machines (SVM) for predicting target packets using packet side channels showed that the best HMM false positive rate (FPR) was much lower than that of SVMs (1.90\% versus 20.27\%). These attacks show that time-sensitive protocols are especially vulnerable to attackers. Defensive transformations may selectively enforce policy on protocols or terminate invalid connections. However, these operations must preserve the underlying protocol's performance and allow desired behavior. By using pre-compiled access control truth tables with firewalls on ROS autonomous fleet communications, we show that attributed-based zero trust access policies can be enforced with minimal network impact with an average 10.5 ms latency. Additionally, policy consistency and protocol verification allowed controlling and tuning induced behavior caused by the transformation. Network transformations often result in performance loss, including cases that conceal or control traffic. However, transformations that split a protocol across multiple paths or sessions mitigate these effects and improve goodput by as much as 30.5\%.