Java serialization for clearing house systems

Clearing houses play a critical role in the infrastructure of financial trad-ing, ensuring reliability. These systems handle millions of transactionevery day. Given the immense volume of messages, an efficient seri-alizer is essential to improve system throughput. This thesis evaluatedMsgCodec, a serializer used in Nasdaq’s clearing house systems. De-veloped by Cinnober, MsgCodec remains relatively undocumented inthe public domain. This thesis was divided into two parts, first partconsisted of analyzing MsgCodec’s serialization process and potentialfactors that can limit throughput. The second part, evaluated MsgCodecperformance relative to alternative high-performant serializers.To analyze MsgCodec, a static analysis was conducted that revealedMsgCodec employs runtime code generation by utilizing Java Reflec-tion in conjunction with the ASM bytecode manipulation framework.Decompilation of the generated code revealed potential performance is-sues. It revealed that MsgCodec relies on Reflection for private fields,which will impact performance. Furthermore, it was revealed that usingOutputStream as the buffer instead of ByteArrayBuf or ByteBufferBufintroduces recursion and deep copying in the serialization process, re-sulting in unnecessary processing.In the second part of the thesis, FlatBuffers and Apache Fury wereselected as suitable alternatives. MsgCodec was benchmarked againstthem using the JMH library. The primary focus was on throughput mea-surements. The results show trade-offs in performance characteristics.However, Apache Fury ultimately emerged as the most promising seri-alizer for Clearing house systems, offering the highest throughput in the scenarios tested.