The limits of knowing: An equilibrium between the observational gain and chaotic loss of information in data assimilation

Abstract Data assimilation (DA) is the most advanced tool for assessing the state of time‐evolving chaotic systems. An estimate (analysis) is derived by combining information from the most recent and previous batches of observations, the latter of which are carried forward in time by first‐guess (FG) forecasts started from previous analyses. Error variance in successful DA cycles fluctuates around an expected value. What factors determine this value? Three parameters are found to determine the level of analysis error variance () or the amount of information in state estimates: information extracted from the most recent set of observations by a DA system (), the growth rate of error in FG (), and the relative weight used for combining information from the latest observations and the FG (). Our key recognition is that in DA systems with stationary performance, information gain from the most recent observations, and information loss due to chaotic error growth, in an expected sense, must be equal. Exploiting this equilibrium relationship, analysis error variance can be expressed as a function of the three driving parameters. Analysis information linearly and exponentially depends on and , respectively, while the optimal weight is a simple function of the error growth rate. An evaluation of four operational DA systems from year 2008 reveals that their quality is driven by the amount of observational information they each extract from a virtually common set of globally available observations. The European Center for Medium‐range Weather Forecast analysis demonstrates the lowest error variance among all evaluated operational systems. However, its analysis quality shows the least sensitivity to the tuning of weight . The error equilibrium relationship reveals that a simple global adjustment of the relative weight between observations and FG may yield an up to 11%–43% reduction in error variance at the different centers.