Data Science Stack Exchange
Data Science Stack Exchange

Questions tagged [markov-process]

A Markov process is a stochastic process for which the Markov property holds: If you know the current state, then the next state is independent of all past states.

Overview

A Markov process is any stochastic process $Y_{t}$ such that the future is conditionally independent of the past, given the present; the distribution of the process only depends on where the process is, not where it has been: $$ P(Y_{t+1}=y_{t+1} |Y_t = y_{t}, Y_{t-1} = y_{t-1}, ..., Y_{1} = y_{1}) = P(Y_{t+1}=y_{t+1} |Y_t = y_{t}) $$ This property is known as the Markov property.

References

The following threads on math.se provide references to resources on Markov processes:

72 questions
20
votes
5 answers

Python library to implement Hidden Markov Models

What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before. Alternatively, is there a more direct approach to performing a time-series analysis…
yad
  • 1,773
  • 3
  • 16
  • 27
17
votes
1 answer

Visualization of multiple Markov models

I am working on a project where we compare over 10 different Markov models, each representing a different treatment plan. Most often single models are visualized with a decision tree or transition state diagram. However, with multiple different…
Andrew Brown
  • 279
  • 1
  • 3
7
votes
3 answers

How do I choose a discount factor in Markov Decision Problems?

I'm referring to the gamma in the Value function:
Austin Capobianco
  • 483
  • 1
  • 4
  • 18
7
votes
1 answer

Comparing transition matrices for Markov chains

I have a population, each unit of which exists in one of several states that change over time. I am using first-order Markov chains to model these state transitions. My population can be segmented into various subpopulations of interest. I've…
R Hill
  • 1,095
  • 10
  • 19
7
votes
2 answers

Reward dependent on (state, action) versus (state, action, successor state)

I am studying reinforcement learning and I am working methodically through Sutton and Barto's book plus David Silver's lectures. I have noticed a minor difference in how the Markov Decision Processes (MDPs) are defined in those two sources, that…
Neil Slater
  • 28,338
  • 4
  • 77
  • 100
4
votes
1 answer

Markov Chains: How much steps to conclude a Transition Matrix

I have just learned Markov Chains which I am using to model a real world problem. The model comprises 3 states [a b c]. For now I am collection data and calculating transitional probabilities:- T[a][b] = #transitions from a to b / #total transitions…
Mangat Rai Modi
  • 569
  • 1
  • 5
  • 10
4
votes
1 answer

Artificially increasing frequency weight of word ending characters in word building

I have a database of letter pair bigrams. For example: +-----------+--------+-----------+ | first | second | frequency | +-----------+--------+-----------+ | gs | so | 1 | | gs | sp | 2 | | gs | sr …
Matt
  • 141
  • 2
4
votes
1 answer

Reinforcement Learning - How are these state values in MRP calculated?

This is a question from the book an Introduction to RL, page 125, example 6.2. The example compares the prediction abilities of TD(0) and constant $ \alpha $ MC when applied to the below Markov reward process (the image is copied form the book):…
Melanie A
  • 167
  • 4
4
votes
1 answer

What is the relationship between MDP and RL?

What is the relationship between Markov Decision Processes and Reinforcement Learning? Could we say RL and DP are two types of MDP?
user10296606
  • 1,784
  • 5
  • 17
  • 31
4
votes
3 answers

Markov Chains for sequential data

I am new to Markov chains and HMM and I am looking for help in developing a program (in python) that predicts the next state based on 20 previous states (lets say 20 states in last 20 months). I have a sequential dataset with 50 customers i.e. the…
mlgal55
  • 43
  • 1
  • 4
4
votes
1 answer

Viterbi-like algorithm suggesting top-N probable state sequences implementation

Traditional Viterbi algorithm (say, for hidden Markov models) provides the most probable hidden state sequence given a sequence of observations. There probably is an algorithm for decoding top-N probable hidden states sequences (k-shortest paths or…
Anton
  • 243
  • 2
  • 10
4
votes
2 answers

Markov switching models

What are some reference sources for understanding Markov switching models?
user16103
  • 43
  • 1
  • 4
3
votes
1 answer

emission probability using hmmlearn package in python

I am learning hmm and try to implement it in Python hmmlearn package(http://hmmlearn.github.io/hmmlearn/hmm.html#building-hmm-and-generating-samples). However I am not quite understand what the documentation says: Classes in this module include…
epx
  • 131
  • 1
  • 3
3
votes
1 answer

If I use Gibbs sampling with a Bayesian model, what do I have to check is memoryless?

Right now I am trying to better understand how Bayesian modeling works with just the basics. I found through reading tutorials that some very basic Bayesian models like Bayesian Hierarchical Modeling use something called the "Gibbs sampling…
pierround
  • 61
  • 4
3
votes
1 answer

When to stop calculating values of each cell in the grid in Reinforcement Learning(dynamic programming) applied on gridworld

Considering application of Reinforcement learning(dynamic programming method performing value iteration) on grid world, in each of the iteration, I go through each of the cell of the grid and update its value depending on its present value and the…
girl101
  • 1,161
  • 2
  • 11
  • 25
1
2 3 4 5