Encoding Data Formats for Evolvability

Abstract—Modern applications change over time, with more features modified or added as new products are launched and existing features are improved. Applications should aim to build systems that make it easy to adapt to changes. In most cases, a change to an application’s features also requires a change in the data it stores, and a new field or record type must be captured or perhaps existing data must be presented in a new way. In order for the system to continue running smoothly, we need to maintain compatibility in both directions – backward compatibility and forward compatibility. Backward compatibility is normally not hard to achieve as an author of the newer code knows the format of data written by the older code, and so user can explicitly handle it (if necessary by simply keeping the old code to read the old data). Forward compatibility can be trickier because it requires older code to ignore additions made by a newer version of the code. In this paper, we will deep dive into formats for encoding data and look into how they handle schema changes and how they support systems where old and new data and code need to coexist. We will then discuss how those formats are used for data storage and for communication in web services, representational state transfer (REST) [2], and remote procedure calls (RPC) [3], as well as message-passing systems such as actors and message queues Keywords—Encoding, Decoding, Serialization, Deserialization, Forward Compatibility, Backward Compatibility, Evolvability, Performance, Compact