Is there an example on how to work with generalized mixture models?

I am working on a simplest complex example to record my learning journey here.

Working with Normal Mixtures works fine, but I’m not quite sure how to debug the error that I’m getting at the bottom of the notebook when trying to do generalized Mixtures. Am I setting up the model incorrectly?

The correct syntax is changing:
components = pm.Poisson('components', mu=lam, shape=(2,))
to
components = pm.Poisson.dist(mu=lam, shape=(2,))

Also, careful of label switching in this case. More information see https://gist.github.com/junpenglao/4d65d1a9bf80e8d371446fadda9deb7a

Thanks @junpenglao! I must have missed the .dist(...) thing somewhere. Was this documented in the docs? Perhaps I can put in a PR to show how this syntax gets used?

Btw, I got the Poisson Mixture working! Thanks for your help! :smile:

I think it is in the doc somewhere but you are right, it is not very clear from the docstring. It would be great to add an Example into the docstring like what you are adding for Normal etc!

1 Like

Thanks @junpenglao! One thing I’m finding is I’m getting errors with putting together Weibull mixtures. Might you have an intuition as to what’s happening? I updated my mixture-model notebook (https://github.com/ericmjl/bayesian-analysis-recipes/blob/master/notebooks/mixture-model.ipynb), and it can be found at the bottom.

The error message is two-fold:

  • No modes.
  • Object not iterable.

Neither makes sense to me, as I’m following the pattern above for the Poisson mixtures.

It might not work directly out of the box… There are some problem with the mixture model in terms of dimension etc, for example:

There are some work needs to be done there…

I noticed that the Weibull distribution doesn’t have a mode defined. Is that because of the definition of the Weibull mode being conditionally dependent on the value of the alpha parameter? (https://en.wikipedia.org/wiki/Weibull_distribution)

Can we have conditional modes in PyMC3 distributions, or do they have to be analytical, non-conditional modes?

Thanks for the great notebook @ericmjl! It was a great help for me. I tried to extend it with other mixtures, e.g. binomial but could not get it to work.

My artificial data is

comp1 = np.random.binomial(n=7, p=.1, size=(1000,)) # - 2
comp2 = np.random.binomial(n=7, p=.7, size=(500,)) # + 4

mix = np.concatenate([comp1, comp2])

and my model speficitation

with pm.Model() as binomial:
    n = pm.Geometric('n', p=1/7)
    p = pm.Uniform('p', lower=.5, upper=1.)
    w = pm.Dirichlet('w', a=np.array([1,1]))
    
    comps = pm.Binomial.dist(n=n, p=p, shape=(2,))
    
    mixture = pm.Mixture('mixture', comp_dists=comps, w=w, observed=mix)

However, when I try to sample from the model,

with binomial:
    trace = pm.sample(1000, chains=2, cores=2)

my notebook crashes. What am I doing wrong here?

could be a memory problem - could you paste the crash message from terminal?
Also, try:

with binomial:
    trace = pm.sample(1000, chains=2, cores=1)

The error message I get is:

You can find the C code in this temporary file: C:\Users\FABIAN~1\AppData\Local                                                                    \Temp\theano_compilation_error_z64rcaov
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\fabianmoss\Anaconda3\lib\multiprocessing\spawn.py", line 105,                                                                     in spawn_main
    exitcode = _main(fd)
  File "C:\Users\fabianmoss\Anaconda3\lib\multiprocessing\spawn.py", line 115,                                                                     in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\compile\function                                                                    _module.py", line 1082, in _constructor_Function
    f = maker.create(input_storage, trustme=True)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\compile\function                                                                    _module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\link.py", li                                                                    ne 699, in make_thunk
    storage_map=storage_map)[:3]
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\vm.py", line                                                                     1091, in make_all
    impl=impl))
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\op.py", line                                                                     955, in make_thunk
    no_recycling)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\op.py", line                                                                     858, in make_c_thunk
    output_storage=node_output_storage)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1217, in make_thunk
    keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1157, in __compile__
    keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cmodule.py",                                                                     line 1189, in module_from_key
    module = lnk.compile_cmodule(location)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1523, in compile_cmodule
    preargs=preargs)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cmodule.py",                                                                     line 2396, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: ('The following error happened while compiling the node', Softmax(El                                                                    emwise{sub,no_inplace}.0), '\n', 'Compilation failed (return status=3): ', '[So                                                                    ftmax(<TensorType(float64, matrix)>)]')
forrtl: error (200): program aborting due to control-C event

To be clear: I did not press CTRL+C :slight_smile:

I tried your suggestion and it kind of worked. Could you explain me why?

Also, for binomial.check_test_point() I get

n                   -1.95
p_interval__        -2.41
w_stickbreaking__   -1.39
mixture               NaN
Name: Log-probability of test_point, dtype: float64
Name: Log-probability of test_point, dtype: float64

The result looks odd though:

If I replace n with a constant value in pm.Binomial.dist I don’t get a NaN value for mixture.

Are you on windows?

Yes I am.

There is a known recurrent issue that happens on windows when sample is run with multiple cores. The problem is not actually restricted to windows. Unix OSes don’t show it so often because the multiprocessing backend defaults to fork, which behaves better than spawn and forkserver which are the available backends on windows.

The errors are very diverse but your particular issue looks strongly related to this and this issues. It is clearly a theano related problem in which the spawned process for some reason that we don’t understand yet, fails to compile the model’s logp (what actually happens is that it tries to unpickle a theano.function and to do so, it tries to recompile it in the new process). The only work around we have for now is to run sample with cores=1.

Thanks a lot! I will check out the issues you mention. Is there anything I can test to help figuring out what’s going wrong with sample on my machine? Also, is this likely to disappear with PyMC4 being based on TensorFlow?

I think that the particular problem with multiprocessing will disappear in pymc4, because we could rely on tensorflow itself to distribute the runs across many devices. We are still far from addressing this, however.
You are, of course, more than welcome to try to explore the root cause of your problematic model. I never managed to reproduce those errors, and I honestly don’t know what is going wrong on the theano side.

1 Like