Adaptive Control and Parameter Estimation in Markov Chains: A Quadratic Case

Abstract

The objective of this thesis is to extend results to a new quadratic case in support of a collectionof existing results in adaptive control and parameter estimation of Markov chains. In the first chapter, we introduce Markov chains and discuss some of their important properties. This chapter will help the reader understand the characteristics of Markov chains, which will be useful in later chapters discussing results on adaptive control of these stochastic processes. In the second chapter, we introduce Martingales and discuss some of their important properties. Martingales are important tools used in the methods of parameter estimation of Markov chains in the later chapters. In the third chapter, we focus on adaptive control of Markov chains. First, we consider controlled Markov chains and some general connections with martingales, namely the Law of Large Numbers and the Central Limit Theorem of Markov chains. Then, we introduce the adaptive control environment with a controlled Markov chain with an unknown parameter. It is after this that we discuss previous important results in [6],[1], and [3] on adaptive control and parameter estimation of Markov chains. Then in the fourth chapter, we outline the processes used and results attained in [8] for a linear case of adaptive control of Markov chains, which we then extend to a quadratic case in the final chapter. For the main results, we perform the process discussed in [7] for the following problem. We consider a controlled Markov chain with a finite state space, whose transition probabilities are assumed to depend quadratically on an unknown real parameter α. Particularly, we study the behavior of the maximum likelihood estimate of α at each time n as n increases under an arbitrary realizable control. We show that the results of [8] extend to the quadratic case with a few additional assumptions. These results are that the sequence of estimates of α converge almost surely, though not necessarily to the true parameter. We characterize those realizations for which convergence does not lead to the true value, and suggest corrections to the control to attain convergence to the true value. In support of previous results, we show that the maximum likelihood estimate converges to a value α∗ indistinguishable from the true value under a control feedback law induced by α∗.