Adiabatic Markov Decision Process: Convergence of Value Iteration Algorithm Markov Decision Process (MDP) is a well-known framework for devising the optimal decision making strategies under uncertainty. Typically, the
decision maker assumes a stationary environment which is characterized by a time-invariant transition probability matrix. However, in many real-world scenarios, this assumption is not
justified, thus the optimal strategy might not provide the expected performance. In this thesis, we study the performance of the classic Value Iteration (VI) algorithm for solving an MDP problem
under non-stationary environments. Specifically, the non-stationary environment is modeled as a sequence of time-variant transition probability matrices governed by an adiabatic evolution inspired
from quantum mechanics.

We characterize the performance of the VI algorithm subject to the rate of change of the underlying environment. The performance is
measured in terms of the convergence rate to the optimal average reward. We show two examples of queuing systems that make use of our analysis framework.