Dynamic decision making under uncertainty

Real-world decision-making rarely occurs as a single, fully informed choice. This thesis tackles three challenging problems of sequential decision-making under uncertainty, where the available information may be imperfect or incomplete. Chapter 2 considers a periodic-review, multi-echelon inventory system with an unknown demand distribution. We propose a non-parametric algorithm that generates a sequence of adaptive ordering decisions based on the stochastic gradient descent method and historical demand data. We compare the T-period cost of our algorithm to that of the clairvoyant, who knows the underlying demand distribution in advance, and we prove that the expected T-period regret is at most O(√T), matching a lower bound for this problem. chapter 3 studies the optimal maintenance of a machine whose state is unobservable and evolves according to a discrete-time, finite-state hidden Markov process. Self-announcing failures trigger immediate corrective replacement; otherwise, the decision-maker chooses among production, inspection, and preventive replacement. We adopt the dual formulation of partially observable Markov decision processes. We characterize optimal policies through graphs with absorbing cycles. We develop a policy iteration algorithm that computes the optimal solution exactly in finite time. We also show that the machine maintenance problem is mathematically equivalent to an inventory management problem with unobserved inventory, thereby revealing a connection in the literature on machine maintenance and inventory control. Chapter 4 evaluates the potential impact of converting non-O donor organs to blood type O in kidney paired donation (KPD) programs. Using data from the Canadian Transplant Registry, we combine mathematical optimization with Monte Carlo simulations to model kidney allocation under stochastic arrivals of candidates and donors to assess how varying levels of blood type conversion affect the number of transplants over a 10-year horizon. Our results show that converting organs from 15% of non-O donors is sufficient to achieve most of the benefits. Improvements are observed across most blood groups, particularly for blood type O, and across all sensitization levels, demonstrating that organ blood type conversion is a promising strategy for improving both efficiency and access in KPD programs.