Abstract: This book provides an accessible introduction to the theory, applications and algorithms of Markov decision processes (MDPs). It delves into the mathematical and algorithmic foundations of decision-making under uncertainty, focusing on problems where trade-offs, uncertainty and dynamic considerations are paramount. By clearly describing the core principles of these decision problems, the book aims to equip readers with the tools to formulate, analyze and solve a wide range of sequential decision problems. The book consists of three parts, each comprised of several chapters. The first part establishes the fundamentals of MDPs, in particular defining the basic model components and optimality criteria, and describing and rigorously formulating a diverse set of applications. The second part covers classical MDP models including finite horizon MDPs, infinite horizon MDPs under discounted, total and average reward optimality criteria, and partially observable MDPs. The final part focuses on approximate dynamic programming and reinforcement learning.  It describes the behavioral foundations of RL and its main methods.