A Markov chain is a mathematical model for a system that moves between states over time, where the next state depends only on the current state and not on the full history.
This is the discrete-state version of the Markov property: the process is “memoryless” once the present state is known.
Core idea
At each step, the system is in some state
So a Markov chain combines:
- a set of possible states
- a rule for moving between states
- a probability distribution over the next state given the current one
This makes it a simple model for random evolution over time.
Transition probabilities
The behavior of a Markov chain is usually summarized by a transition matrix
That connects Markov chains to Matrices, because repeated steps can be studied using matrix operations such as Matrix Multiplication.
If you know the current distribution over states, multiplying by the transition matrix gives the distribution after one more step.
Why it matters
Markov chains are used when:
- the system evolves in steps
- the future is uncertain
- the current state contains all the relevant information for prediction
They are a foundation for more advanced models in probability, statistics, reinforcement learning, and control.
Connection to MDPs
A Markov decision process extends a Markov chain by adding actions and rewards.
In a plain Markov chain, the transitions happen according to fixed probabilities.
In an MDP, the transition probabilities depend on the action chosen by an agent.