Discrete Stochastic Processes

Discrete stochastic processes are essentially probabilistic systems that evolve in time via random changes occurring at discrete fixed or random intervals. This course aims to help students acquire both the mathematical principles and the intuition necessary to create, analyze, and understand insightful models for a broad range of these processes. The range of areas for which discrete stochastic-process models are useful is constantly expanding, and includes many applications in engineering, physics, biology, operations research and finance.

Syllabus

Course Meeting Times

Lectures: 2 sessions / week, 1.5 hours / session

Course Description

Discrete stochastic processes are essentially probabilistic systems that evolve in time via random changes occurring at discrete fixed or random intervals. This course aims to help students acquire both the mathematical principles and the intuition necessary to create, analyze, and understand insightful models for a broad range of these processes. The range of areas for which discrete stochastic-process models are useful is constantly expanding, and includes many applications in engineering, physics, biology, operations research and finance.

Prerequisites

Thorough understanding of elementary probability at the level of , which uses the following text:

Bertsekas, Dimitri, and John Tsitsiklis. . 2nd ed. Athena Scientific, 2008. ISBN: 9781886529236.

Some patience and affinity for careful mathematical reasoning.

Video Lectures

Lecture 1: Introduction and Probability Review

Description: Probability, as it appears in the real world, is related to axiomatic mathematical models. Events, independence, and random variables are reviewed, stressing both the axioms and intuition.

Instructor: Prof. Robert Gallager

Description: The review of probability is continued with expectation, multiple random variables, and conditioning. We then move on to develop the weak law of large numbers (WLLN) and the Bernoulli process.

Instructor: Prof. Robert Gallager

Lecture 3: Law of Large Numbers, Convergence

Description: This lecture begins with the use of the WLLN in probabilistic modeling. Next the central limit theorem, the strong law of large numbers (SLLN), and convergence are discussed.

Instructor: Prof. Robert Gallager

Lecture 4: Poisson (The Perfect Arrival Process)

Description: This lecture begins with a description of arrival processes, and continues on to describe the Poisson process from three different viewpoints.

Instructor: Prof. Robert Gallager

Lecture 5: Poisson Combining and Splitting

Description: In this lecture, many problem solving techniques are developed using, first, combining and splitting of various Poisson processes, and, second, conditioning on the number of arrivals in an interval.

Instructor: Prof. Robert Gallager

Lecture 6: From Poisson to Markov

Description: This lecture treats joint conditional densities for Poisson processes and then defines finite-state Markov chains. Recurrent and transient states, periodic states, and ergodic chains are discussed.

(Courtesy of Mina Karzand. Used with permission.)

Instructor: Mina Karzand

Lecture 7: Finite-state Markov Chains; The Matrix Approach

Description: The transition matrix approach to finite-state Markov chains is developed in this lecture. The powers of the transition matrix are analyzed to understand steady-state behavior.

(Courtesy of Shan-Yuan Ho. Used with permission.)

Instructor: Shan-Yuan Ho

Lecture 8: Markov Eigenvalues and Eigenvectors

Description: This lecture covers eigenvalues and eigenvectors of the transition matrix and the steady-state vector of Markov chains. It also includes an analysis of a 2-state Markov chain and a discussion of the Jordan form.

Instructor: Prof. Robert Gallager

(Many more)