Solving overall risk minimization by stationary policy heuristics

Abstract We propose a heuristic method for solving the overall risk minimization problem, specifically the infinite-horizon dynamic programming problem, which uses a coherent risk measure of the overall discounted reward as its criterion. The original problem is very complex, and its optimal policies may depend on the entire history of previous decisions. The heuristic consists in restricting attention to Markov policies—those depending only on the current state—and in consecutive local approximations of the criterion, a static coherent risk measure, by its nested dynamic counterpart: a limit nested risk measure. The approximating problems with the nested measures can be reformulated using the Bellman equation and, as such, can be solved by standard algorithms. We introduce several variants of this heuristic. Consequently, we conduct three numerical experiments, all of which demonstrate the efficiency of our heuristic in finding effective solutions. Due to its speed, the heuristic is suitable for finding “good enough” solutions, which may be further refined.