Predicting Recurrent Events with an Adaptive Non-Markov Memory Model

It is well known that many sequential events are not completely random: The past can influence the future. Events may thus follow cycles, cluster after earlier events, repeat after a characteristic delay or change behaviour when the underlying system enters a different state. Because such relationships recur, the historical record can carry information about the probability of future events. Most forecasting methods restrict how that information is used, for example by weighting only recent observations, assuming a fixed decay rate of memory or fitting a predetermined model of dependence that is then held fixed.This paper introduces adaptive non-Markov memory (ANM), a forecasting framework that does not decide in advance which part of the past matters. ANM retains the complete observed history, represents it at multiple timescales and searches it for repeating cycles, delayed recurrences, changes of regime and similar past states. After every time step, whether or not an event occurs, it updates how much weight each form of historical evidence receives. When a continuous measurement of the underlying process is available, ANM also compares its recent trajectory with earlier ones and forecasts from what followed the most similar past states.ANM was tested, in stages fixed before they were run, on 22 various synthetic processes with known structure, seven nonlinear and stochastic control systems and 15 real event series. Its advantages in forecasting accuracy depended on whether the past contained recoverable information about the future. It captured 93–95% of the achievable gain on strongly cyclic processes and on five chaotic systems the continuous measurement reduced forecast error (log-loss) by a further 52–90%. On real data it improved on standard renewal and ETAS baselines by 15–93% on sunspots, air pollution, electricity demand and two temperature series and stayed within 3% of them on rainfall, tree rings, coal-mine disasters and seven financial series. It was not significantly different from GARCH on market losses. Against neural Hawkes-type models it was significantly better on four series, significantly worse on three and not significantly different on eight. Of the pre-specified real-data criteria, only the GARCH comparison was met.The central result is that past observations improve forecasts when a process repeatedly leaves identifiable patterns in its history and those patterns need not occur at a single predetermined timescale. However, retaining more historical information does not add predictability when the process contains little recoverable structure.