You could try quantecon.MarkovChain if that’s all you need to do, see the lecture I linked from them above.
quantecon.MarkovChain