Analysis of Hybrid Digital Signatures for Post Quantum Authentication in Mbed TLS

Post-quantum (PQ) cryptography bases its security on mathematical problems believed to be hard even with respect to quantum computers. This thesis evaluates the performance of the Transport Layer Security (TLS) handshake when using hybrid PQ digital signatures that combine two or three PQ schemes. A signature combiner supporting 17 standardized or NIST-proposed schemes is implemented, enabling both concatenation and strong nesting for an arbitrary number of signatures. The Mbed TLS library is extended to integrate this combiner into its handshake process. Benchmarking a wide range of hybrid configurations reveals that public key and signature sizes are major bottlenecks. Practical size limits for handshake messages in TLS implementations and imbalances in scheme performances severely restrict the set of usable hybrids. However, a small number of two- and three-scheme hybrids appear to be relatively viable. While there are two-scheme hybrids with only marginal performance overheads relatively suitable for TLS, combining three or more schemes leads to more significant performance overheads. While these overheads may be acceptable depending on the application, larger hybrids are deemed to be more immediately suited for use cases with infrequent, long-duration connections, such as IKEv3 or SSH.