Efficient Data Confidence Fabrics with Compact Annotations

The need to trust data has become a key requirement in modern distributed systems. To facilitate measurable trust and confidence in data and applications spanning heterogeneous systems, the emerging concept of a Data Confidence Fabric (DCF) offers a compelling solution. Data producers and processors provide metadata, known as annotations, recording trust insertion along the data delivery chain. Thus, it is possible to asses the trustworthiness of data before processing it. While a DCF, such as the Alvarium framework, enables this management of trust, there is a cost in terms of the overheads associated with security annotations themselves. To improve efficiency of a DCF, we therefore propose a set of techniques for making annotations more compact and to reduce the number of DCF transactions. Our work shows that transaction efficiency gains of up to 93% in our considered use cases can be achieved.