Adaptive nonstationary value iteration for discounted control of Piecewise Deterministic Markov Processes

Abstract We study the adaptive infinite-horizon discounted control problem for Piecewise Deterministic Markov Processes (PDMPs) using a Nonstationary Value Iteration (NVI) scheme. PDMPs, as introduced by Davis (Markov models and optimization, monographs on statistics and applied probability, Chapman and Hall, London, 1993) evolve deterministically between random jumps whose jump rate $$\lambda $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>λ</mml:mi> </mml:math> , transition measure Q , and cost C depend on an unknown parameter $$\beta ^*$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mi>β</mml:mi> <mml:mo>∗</mml:mo> </mml:msup> </mml:math> . The proposed NVI algorithm recursively updates the value function using current parameter estimates, enabling online implementation. We show that, for any sequence of strongly consistent estimators $$\{\beta ^*_n\}$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mo>{</mml:mo> <mml:msubsup> <mml:mi>β</mml:mi> <mml:mi>n</mml:mi> <mml:mo>∗</mml:mo> </mml:msubsup> <mml:mo>}</mml:mo> </mml:mrow> </mml:math> converging almost surely to $$\beta ^*$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msup> <mml:mi>β</mml:mi> <mml:mo>∗</mml:mo> </mml:msup> </mml:math> , the resulting policy is asymptotically optimal under the discounted criterion.