Generating samples using mcmc method from PyMc

There are several problems with this model. A PyMC model is not normal python code. Instead, it lazily constructs a computational graph. This graph represents, but does not compute!, the expressions you write down. As a result, something like this block:

        if states[i] == target_state:
            transition_count +=1

is not going to work. In general, loops are never advised. There is a looping construct in PyMC called scan, if you need to do recursive computation. In general, though, scan is like a nuclear bomb, and there are often simpler solutions to explore before you go down that road.

In your case, you are trying to estimate a discrete markov chain, and we have a probability distribution for that in pymc-experimental. You can have a look at the first example in this notebook for a model that seems to be exactly what you’re trying to accomplish here.

One other issue is that you cannot call pm.sample then check a model variable like transition_count. You need to add the symbolic computation to the output of pm.sample using pm.Deterministic. In your case, though, you aren’t estimating anything, because the transition matrix is fixed. If you sample then compute shares of hidden states transitions from state i to state j, it will simply return transition_matrix[i,j]. See the above notebook for an example of using a random variable to estimate a transition matrix.