Hi @SCon.
So I think that the thread you link to (and Pomegranate) are a distraction. I believe everything you want to do here can be done in PyMC.
It really sounds like you are describing Signal Detection Theory, so you are basically doing psychophysics here
Take a look at Implementing normal CDF for signal detection theory model (includes full sample code).
Note that in signal detection theory they use terms like
- “hit rate” which is the same as a “correct detection” (1-alpha) and
- “miss rate” which is the same as a Type II error / false negative (beta)
See Type I and type II errors - Wikipedia for more on this if it’s not clear.
Feel very free to shout if this doesn’t solve your problem.