Is there an example on how to work with generalized mixture models?

ericmjl · January 12, 2018, 9:03pm

I am working on a simplest complex example to record my learning journey here.

ericmjl/bayesian-analysis-recipes/blob/master/mixture-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gaussian Mixtures\n",
    "\n",
    "Sometimes, our data look like they are generated by a \"mixture\" model. What do we mean by that? In statistics land, it means we believe that there are \"mixtures\" of subpopulations generating the data that we observe. A common activity, then, is to estimate the subpopulation parameters.\n",
    "\n",
    "Let's take a look at it by generating some simulated data to illustrate the point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {

This file has been truncated. show original

Working with Normal Mixtures works fine, but I’m not quite sure how to debug the error that I’m getting at the bottom of the notebook when trying to do generalized Mixtures. Am I setting up the model incorrectly?

junpenglao · January 12, 2018, 9:59pm

The correct syntax is changing:
components = pm.Poisson('components', mu=lam, shape=(2,))
to
components = pm.Poisson.dist(mu=lam, shape=(2,))

Also, careful of label switching in this case. More information see https://gist.github.com/junpenglao/4d65d1a9bf80e8d371446fadda9deb7a

ericmjl · January 14, 2018, 3:53am

Thanks @junpenglao! I must have missed the .dist(...) thing somewhere. Was this documented in the docs? Perhaps I can put in a PR to show how this syntax gets used?

Btw, I got the Poisson Mixture working! Thanks for your help!

junpenglao · January 14, 2018, 8:10am

I think it is in the doc somewhere but you are right, it is not very clear from the docstring. It would be great to add an Example into the docstring like what you are adding for Normal etc!

ericmjl · January 17, 2018, 3:51pm

Thanks @junpenglao! One thing I’m finding is I’m getting errors with putting together Weibull mixtures. Might you have an intuition as to what’s happening? I updated my mixture-model notebook (https://github.com/ericmjl/bayesian-analysis-recipes/blob/master/notebooks/mixture-model.ipynb), and it can be found at the bottom.

The error message is two-fold:

No modes.
Object not iterable.

Neither makes sense to me, as I’m following the pattern above for the Poisson mixtures.

junpenglao · January 17, 2018, 4:49pm

It might not work directly out of the box… There are some problem with the mixture model in terms of dimension etc, for example:

github.com/pymc-devs/pymc

Multinomial Mixture in PyMC3

opened 09:18PM - 04 Aug 17 UTC

closed 12:22PM - 22 Dec 18 UTC

vsmolyakov

I'm trying to create a mixture of multinomials using PyMC using `pm.Mixture` con…struct as described in [documentation](https://pymc-devs.github.io/pymc3/api/distributions/mixture.html). However, I'm running into dimension mismatch error: ``` T = 4 V = 10 with pm.Model() as model: H = pm.Dirichlet("H", a=np.ones(V), shape=V) phi = [pm.Multinomial('phi_%s' %t, n=10, p=H, shape=V) for t in range(T)] phi_dists = [comp.distribution for comp in phi] G0 = pm.Mixture('G0', w = np.ones(T)/T, comp_dists = phi_dists) ``` This code results in the following error: `ValueError: Input dimension mis-match. (input[0].shape[1] = 4, input[1].shape[1] = 10)` I do have a mixture with 4 components each of size 10. I also tried constructing the mixture manually: ``` with model: phi_mix = [w[t]*phi_top[t] for t in range(T)] G0 = pm.Deterministic('G0', sum(phi_mix)) ``` However, the above code lead to a different downstream error. So I was wondering how I can construct a mixture of multinomials using PyMC3? This should be a common task for discrete data models. This question is also posted on [stack overflow](https://stackoverflow.com/questions/45495241/mixture-of-multinomials-in-pymc3).

github.com/pymc-devs/pymc

Mixture distribution does not play well with `size` parameter

opened 02:40PM - 05 Oct 17 UTC

closed 12:24PM - 22 Dec 18 UTC

ColCarroll

bug

## Description of your problem From https://stackoverflow.com/questions/46581…700/pymc3-sampling-multiple-numbers-from-normal-mixture-distribution-doesnt-work See the example below **Please provide a minimal, self-contained, and reproducible example.** import pymc3 as pm pm.NormalMixture.dist( w=0.5 * np.ones(2), mu=np.ones(2), sd=np.ones(2)).random(size=2) **Please provide the full traceback.** ``` --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-25-bccf91a988a4> in <module>() 2 w=0.5 * np.ones(2), 3 mu=np.ones(2), ----> 4 sd=np.ones(2)).random(size=2) ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/mixture.py in random(self, point, size, repeat) 133 dist_shape=self.shape, 134 size=size).squeeze() --> 135 comp_samples = self._comp_samples(point=point, size=size, repeat=repeat) 136 137 if comp_samples.ndim > 1: ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/mixture.py in _comp_samples(self, point, size, repeat) 101 def _comp_samples(self, point=None, size=None, repeat=None): 102 try: --> 103 samples = self.comp_dists.random(point=point, size=size, repeat=repeat) 104 except AttributeError: 105 samples = np.column_stack([comp_dist.random(point=point, size=size, repeat=repeat) ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/continuous.py in random(self, point, size, repeat) 260 return generate_samples(stats.norm.rvs, loc=mu, scale=tau**-0.5, 261 dist_shape=self.shape, --> 262 size=size) 263 264 def logp(self, value): ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/distribution.py in generate_samples(generator, *args, **kwargs) 389 prefix_shape, 390 *args, **kwargs) --> 391 return reshape_sampled(samples, size, dist_shape) ~/.venv/sampled/lib/python3.6/site-packages/pymc3/distributions/distribution.py in reshape_sampled(sampled, size, dist_shape) 305 306 if np.size(sampled) == 1 or repeat_shape or dist_shape: --> 307 return np.reshape(sampled, repeat_shape + dist_shape) 308 else: 309 return sampled ~/.venv/sampled/lib/python3.6/site-packages/numpy/core/fromnumeric.py in reshape(a, newshape, order) 230 [5, 6]]) 231 """ --> 232 return _wrapfunc(a, 'reshape', newshape, order=order) 233 234 ~/.venv/sampled/lib/python3.6/site-packages/numpy/core/fromnumeric.py in _wrapfunc(obj, method, *args, **kwds) 55 def _wrapfunc(obj, method, *args, **kwds): 56 try: ---> 57 return getattr(obj, method)(*args, **kwds) 58 59 # An AttributeError occurs if the object does not have ValueError: cannot reshape array of size 4 into shape (2,) ``` **Please provide any additional information below.** It looks like we are using the same `size` parameter in two places where they might mean different things [around here](https://github.com/pymc-devs/pymc3/blob/master/pymc3/distributions/mixture.py#L130): w_samples = generate_samples(random_choice, w=w, broadcast_shape=w.shape[:-1] or (1,), dist_shape=self.shape, size=size).squeeze() comp_samples = self._comp_samples(point=point, size=size, repeat=repeat) ## Versions and main components * PyMC3 Version: `master` * Theano Version: 0.9.0 * Python Version: 3.6 * Operating system: osx * How did you install PyMC3: (conda/pip) `pip`

There are some work needs to be done there…

ericmjl · January 17, 2018, 5:04pm

I noticed that the Weibull distribution doesn’t have a mode defined. Is that because of the definition of the Weibull mode being conditionally dependent on the value of the alpha parameter? (https://en.wikipedia.org/wiki/Weibull_distribution)

Can we have conditional modes in PyMC3 distributions, or do they have to be analytical, non-conditional modes?

fabianmoss · March 14, 2019, 11:30am

Thanks for the great notebook @ericmjl! It was a great help for me. I tried to extend it with other mixtures, e.g. binomial but could not get it to work.

My artificial data is

comp1 = np.random.binomial(n=7, p=.1, size=(1000,)) # - 2
comp2 = np.random.binomial(n=7, p=.7, size=(500,)) # + 4

mix = np.concatenate([comp1, comp2])

and my model speficitation

with pm.Model() as binomial:
    n = pm.Geometric('n', p=1/7)
    p = pm.Uniform('p', lower=.5, upper=1.)
    w = pm.Dirichlet('w', a=np.array([1,1]))
    
    comps = pm.Binomial.dist(n=n, p=p, shape=(2,))
    
    mixture = pm.Mixture('mixture', comp_dists=comps, w=w, observed=mix)

However, when I try to sample from the model,

with binomial:
    trace = pm.sample(1000, chains=2, cores=2)

my notebook crashes. What am I doing wrong here?

junpenglao · March 14, 2019, 2:01pm

could be a memory problem - could you paste the crash message from terminal?
Also, try:

with binomial:
    trace = pm.sample(1000, chains=2, cores=1)

fabianmoss · March 14, 2019, 2:54pm

The error message I get is:

You can find the C code in this temporary file: C:\Users\FABIAN~1\AppData\Local                                                                    \Temp\theano_compilation_error_z64rcaov
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\fabianmoss\Anaconda3\lib\multiprocessing\spawn.py", line 105,                                                                     in spawn_main
    exitcode = _main(fd)
  File "C:\Users\fabianmoss\Anaconda3\lib\multiprocessing\spawn.py", line 115,                                                                     in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\compile\function                                                                    _module.py", line 1082, in _constructor_Function
    f = maker.create(input_storage, trustme=True)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\compile\function                                                                    _module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\link.py", li                                                                    ne 699, in make_thunk
    storage_map=storage_map)[:3]
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\vm.py", line                                                                     1091, in make_all
    impl=impl))
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\op.py", line                                                                     955, in make_thunk
    no_recycling)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\op.py", line                                                                     858, in make_c_thunk
    output_storage=node_output_storage)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1217, in make_thunk
    keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1157, in __compile__
    keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cmodule.py",                                                                     line 1189, in module_from_key
    module = lnk.compile_cmodule(location)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cc.py", line                                                                     1523, in compile_cmodule
    preargs=preargs)
  File "C:\Users\fabianmoss\Anaconda3\lib\site-packages\theano\gof\cmodule.py",                                                                     line 2396, in compile_str
    (status, compile_stderr.replace('\n', '. ')))
Exception: ('The following error happened while compiling the node', Softmax(El                                                                    emwise{sub,no_inplace}.0), '\n', 'Compilation failed (return status=3): ', '[So                                                                    ftmax(<TensorType(float64, matrix)>)]')
forrtl: error (200): program aborting due to control-C event

To be clear: I did not press CTRL+C

I tried your suggestion and it kind of worked. Could you explain me why?

Also, for binomial.check_test_point() I get

n                   -1.95
p_interval__        -2.41
w_stickbreaking__   -1.39
mixture               NaN
Name: Log-probability of test_point, dtype: float64
Name: Log-probability of test_point, dtype: float64

The result looks odd though:

fabianmoss · March 14, 2019, 3:05pm

If I replace n with a constant value in pm.Binomial.dist I don’t get a NaN value for mixture.

lucianopaz · March 14, 2019, 4:53pm

Are you on windows?

fabianmoss · March 14, 2019, 5:05pm

Yes I am.

lucianopaz · March 14, 2019, 9:35pm

There is a known recurrent issue that happens on windows when sample is run with multiple cores. The problem is not actually restricted to windows. Unix OSes don’t show it so often because the multiprocessing backend defaults to fork, which behaves better than spawn and forkserver which are the available backends on windows.

The errors are very diverse but your particular issue looks strongly related to this and this issues. It is clearly a theano related problem in which the spawned process for some reason that we don’t understand yet, fails to compile the model’s logp (what actually happens is that it tries to unpickle a theano.function and to do so, it tries to recompile it in the new process). The only work around we have for now is to run sample with cores=1.

fabianmoss · March 16, 2019, 5:08pm

Thanks a lot! I will check out the issues you mention. Is there anything I can test to help figuring out what’s going wrong with sample on my machine? Also, is this likely to disappear with PyMC4 being based on TensorFlow?

lucianopaz · March 16, 2019, 9:25pm

I think that the particular problem with multiprocessing will disappear in pymc4, because we could rely on tensorflow itself to distribute the runs across many devices. We are still far from addressing this, however.
You are, of course, more than welcome to try to explore the root cause of your problematic model. I never managed to reproduce those errors, and I honestly don’t know what is going wrong on the theano side.

Topic		Replies	Views
Poisson Mixture Model & Label Switching v5	7	1381	August 3, 2022
Weird shape behavior/error using a mixture Questions	0	408	March 2, 2020
When treating a small data, how to choose a model modeling	3	336	June 2, 2023
Modelling a Mixture Model modeling	1	83	June 19, 2024
Mixture Model Latent Variable (augmented-data) example v5	5	426	July 20, 2023

Is there an example on how to work with generalized mixture models?

Related topics