KeyError Setting Mixture Proportions for Mixture Model

Questions
1

I have the following model:

pOn ~ Beta(alpha, beta)
on ~ Bernoulli(pOn)
w = [on, (1 - on)]
tri_giv_on ~ Bernoulli(p)
tri_giv_not_on ~ Bernoulli(q) 
triangle ~ BernoulliMixture(w, [tri_giv_on, tri_giv_not_on])

I’ve tried to implement it using pymc3

with pymc3.Model() as model:
   pOn = pymc3.Beta('pOn', alpha=on_count, beta=(schema_count - on_count));
   on = pymc3.Bernoulli('on', pOn)

   triangle_mixture_weights = [on, (1. - on)]
   tri_giv_on = pymc3.Bernoulli.dist(pTri_given_not_on + tri_delta_on)
   tri_giv_not_on = pymc3.Bernoulli.dist(pTri_given_not_on)
   triangle = pymc3.Mixture('triangle', w=triangle_mixture_weights, \
                               comp_dists=[tri_giv_on, tri_giv_not_on], \
                               shape=2)
   res = pymc3.sample_prior_predictive(1000)

So, the model isn’t perfect. There are some things I can do to make it better, but still,I’m thinking that conceptually, this should work. This, however, isn’t the case. When I run the model, python tells me:

Traceback (most recent call last):
  File "test.py", line 114, in <module>
    res = pymc3.sample_prior_predictive(n)
  File "/usr/local/lib/python2.7/dist-packages/pymc3/sampling.py", line 1325, in sample_prior_predictive
    values = draw_values([model[name] for name in names], size=samples)
  File "/usr/local/lib/python2.7/dist-packages/pymc3/distributions/distribution.py", line 369, in draw_values
    size=size)
  File "/usr/local/lib/python2.7/dist-packages/pymc3/distributions/distribution.py", line 463, in _draw_value
    return param.random(point=point, size=size)
  File "/usr/local/lib/python2.7/dist-packages/pymc3/model.py", line 43, in __call__
    return getattr(self.obj, self.method_name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/pymc3/distributions/mixture.py", line 166, in random
    size=size).squeeze()
  File "/usr/local/lib/python2.7/dist-packages/pymc3/distributions/distribution.py", line 621, in generate_samples
    samples = [generator(*args, **kwargs).reshape(size_tup + (1,)) for _ in range(np.prod(suffix, dtype=int))]
  File "/usr/local/lib/python2.7/dist-packages/pymc3/distributions/dist_math.py", line 322, in random_choice
    size = kwargs.pop('size')
KeyError: 'size'

I’m not exactly sure what this error is about, but python is happy when I replace pOn and on random variables with a new scalar variable on = 1..

NOTE: When i do pymc3.sample(1000) instead of pymc3.sample_prior_predictive(1000), python is happy too.

2

seems to be shape related - @lucianopaz could you take a look?

meanwhile, @dmenager you should upgrade to python3 - py2 is not supported by pymc3 any more and your code might break at some point!

3

Ah, alright! I’ll switch over to python 3!

4

Are you running pymc3.6? If yes, you will have to install the current master branch, which only works on python>=3.5, and retry.

The current master has done many changes to almost all of the involved components of the traceback you posted: we’ve changed Mixture, random_choice and generate_samples. So your error will, in the best scenario, be fixed or, in the worse case, have a different traceback.

5

@lucianopaz, I’m noticing some interesting behavior now that I’ve updated python to 3.5. I installed pymc 3.6 using pip3 install.

The odd behavior I’m observing is that sometimes the code runs, but other times it doesn’t. After running the code two consecutive times, I got:

Traceback (most recent call last):
  File "test.py", line 114, in <module>
    res = pymc3.sample_prior_predictive(n)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/sampling.py", line 1325, in sample_prior_predictive
    values = draw_values([model[name] for name in names], size=samples)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 369, in draw_values
    size=size)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 463, in _draw_value
    return param.random(point=point, size=size)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/model.py", line 43, in __call__
    return getattr(self.obj, self.method_name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/mixture.py", line 152, in random
    w = draw_values([self.w], point=point)[0]
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 400, in draw_values
    size=size)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 508, in _draw_value
    output = np.array([func(*v) for v in zip(*values)])
TypeError: zip argument #2 must support iteration

AND

Traceback (most recent call last):
  File "test.py", line 114, in <module>
    res = pymc3.sample_prior_predictive(n)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/sampling.py", line 1325, in sample_prior_predictive
    values = draw_values([model[name] for name in names], size=samples)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 369, in draw_values
    size=size)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 463, in _draw_value
    return param.random(point=point, size=size)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/model.py", line 43, in __call__
    return getattr(self.obj, self.method_name)(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/mixture.py", line 166, in random
    size=size).squeeze()
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 621, in generate_samples
    samples = [generator(*args, **kwargs).reshape(size_tup + (1,)) for _ in range(np.prod(suffix, dtype=int))]
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/distribution.py", line 621, in <listcomp>
    samples = [generator(*args, **kwargs).reshape(size_tup + (1,)) for _ in range(np.prod(suffix, dtype=int))]
  File "/usr/local/lib/python3.5/dist-packages/pymc3/distributions/dist_math.py", line 322, in random_choice
    size = kwargs.pop('size')
KeyError: 'size'
6

You’re still installing version 3.6, not the current master branch which is the unfinished version 3.7. To install that you need to

pip3 uninstall pymc3
pip3 install git+https://github.com/pymc-devs/pymc3.git
7

When I run the code, I can sample the prior predictive, but when I try to sample the posterior, I get an error related to theano in the sampler:

Multiprocess sampling (2 chains in 2 jobs)
CompoundStep
>Metropolis: [triangle]
>NUTS: [pOn, pos]
>BinaryGibbsMetropolis: [on]
Sampling 2 chains:   0%|                            | 0/3000 [00:00<?, ?draws/s]
pymc3.parallel_sampling.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
TypeError: expected type_num 5 (NPY_INT32) got 7

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 110, in run
    self._start_loop()
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 160, in _start_loop
    point, stats = self._compute_point()
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 191, in _compute_point
    point, stats = self._step_method.step(self._point)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/step_methods/compound.py", line 27, in step
    point, state = method.step(point)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/step_methods/arraystep.py", line 175, in step
    apoint, stats = self.astep(self.bij.map(point))
  File "/usr/local/lib/python3.5/dist-packages/pymc3/step_methods/metropolis.py", line 162, in astep
    accept = self.delta_logp(q, q0)
  File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/usr/local/lib/python3.5/dist-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/lib/python3/dist-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
TypeError: expected type_num 5 (NPY_INT32) got 7
Apply node that caused the error: Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}(InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0}, TensorConstant{(1, 1) of 1})
Toposort index: 22
Inputs types: [TensorType(int32, col), TensorType(int8, (True, True)), TensorType(int8, (True, True))]
Inputs shapes: [(2, 1), (1, 1), (1, 1)]
Inputs strides: [(8, 8), (1, 1), (1, 1)]
Inputs values: [array([[2],
       [0]]), array([[0]], dtype=int8), array([[1]], dtype=int8)]
Outputs clients: [[Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0.0}, TensorConstant{(1, 1) of -inf}, TensorConstant{(1, 1) of -inf}), Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of ..4393873245}, TensorConstant{(1, 1) of ..2804325936}, TensorConstant{(1, 1) of -inf})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
"""

The above exception was the direct cause of the following exception:

TypeError: expected type_num 5 (NPY_INT32) got 7
Apply node that caused the error: Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}(InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0}, TensorConstant{(1, 1) of 1})
Toposort index: 22
Inputs types: [TensorType(int32, col), TensorType(int8, (True, True)), TensorType(int8, (True, True))]
Inputs shapes: [(2, 1), (1, 1), (1, 1)]
Inputs strides: [(8, 8), (1, 1), (1, 1)]
Inputs values: [array([[2],
       [0]]), array([[0]], dtype=int8), array([[1]], dtype=int8)]
Outputs clients: [[Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0.0}, TensorConstant{(1, 1) of -inf}, TensorConstant{(1, 1) of -inf}), Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of ..4393873245}, TensorConstant{(1, 1) of ..2804325936}, TensorConstant{(1, 1) of -inf})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 114, in <module>
    res = pymc3.sample(n)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/sampling.py", line 432, in sample
    trace = _mp_sample(**sample_args)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/sampling.py", line 965, in _mp_sample
    for draw in sampler:
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 393, in __iter__
    draw = ProcessAdapter.recv_draw(self._active)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 297, in recv_draw
    raise error from old_error
RuntimeError: Chain 1 failed.
8

Try setting the dtype like this

If that doesn’t work, it may be related to an issue we’re looking into but haven’t fully identified that slowed our run times.

9

Ok, so I think that I found the cause of the bug here, where the int64 dtype (which is type_num 7) is forced by the step method instead of looking at the variables’ particular dtype (int32 by default). I’ll work on a patch soon but for now, setting on's dtype to int64 should do the trick

10

One other thing, why do you set triangle's shape=2?

11

Well,

I set triangle's shape=2 probably mistakenly. I’ve been trying to understand the shape argument. I was thinking that since I have two mixture components, the mixture model will have shape=2. What I really want is a binary variable where triangle is either 1 or 0.

I just set shape=1, and testval=1., and it seems ok. I still get the error about the chain failing.

12

I applied this change:

n = 1000
with pymc3.Model() as model:
   # On 
   pOn = pymc3.Beta('pOn', alpha=on_count, beta=(schema_count - on_count))
   on = pymc3.Bernoulli('on', p=pOn, dtype="int64")

   # Triangle
   
   triangle_mixture_weights = [on, (1. - on)]
   tri_giv_on = pymc3.Bernoulli.dist(pTri_given_not_on + tri_delta_on)
   tri_giv_not_on = pymc3.Bernoulli.dist(pTri_given_not_on)
   triangle = pymc3.Mixture('triangle', w=triangle_mixture_weights, \
                            comp_dists=[tri_giv_on, tri_giv_not_on], \
                            shape=2)
   res = pymc3.sample(n)

I receive the following error:

Multiprocess sampling (2 chains in 2 jobs)
CompoundStep
>NUTS: [pOn, pos]
>Metropolis: [triangle]
>BinaryGibbsMetropolis: [on]
Sampling 2 chains:   0%|                            | 0/3000 [00:00<?, ?draws/s]
pymc3.parallel_sampling.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
TypeError: expected type_num 5 (NPY_INT32) got 7

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 110, in run
    self._start_loop()
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 160, in _start_loop
    point, stats = self._compute_point()
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 191, in _compute_point
    point, stats = self._step_method.step(self._point)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/step_methods/compound.py", line 27, in step
    point, state = method.step(point)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/step_methods/arraystep.py", line 175, in step
    apoint, stats = self.astep(self.bij.map(point))
  File "/usr/local/lib/python3.5/dist-packages/pymc3/step_methods/metropolis.py", line 162, in astep
    accept = self.delta_logp(q, q0)
  File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/usr/local/lib/python3.5/dist-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/lib/python3/dist-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
TypeError: expected type_num 5 (NPY_INT32) got 7
Apply node that caused the error: Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}(InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0}, TensorConstant{(1, 1) of 1})
Toposort index: 22
Inputs types: [TensorType(int32, col), TensorType(int8, (True, True)), TensorType(int8, (True, True))]
Inputs shapes: [(2, 1), (1, 1), (1, 1)]
Inputs strides: [(8, 8), (1, 1), (1, 1)]
Inputs values: [array([[2],
       [4]]), array([[0]], dtype=int8), array([[1]], dtype=int8)]
Outputs clients: [[Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0.0}, TensorConstant{(1, 1) of -inf}, TensorConstant{(1, 1) of -inf}), Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of ..4393873245}, TensorConstant{(1, 1) of ..2804325936}, TensorConstant{(1, 1) of -inf})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
"""

The above exception was the direct cause of the following exception:

TypeError: expected type_num 5 (NPY_INT32) got 7
Apply node that caused the error: Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}(InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0}, TensorConstant{(1, 1) of 1})
Toposort index: 22
Inputs types: [TensorType(int32, col), TensorType(int8, (True, True)), TensorType(int8, (True, True))]
Inputs shapes: [(2, 1), (1, 1), (1, 1)]
Inputs strides: [(8, 8), (1, 1), (1, 1)]
Inputs values: [array([[2],
       [4]]), array([[0]], dtype=int8), array([[1]], dtype=int8)]
Outputs clients: [[Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of 0.0}, TensorConstant{(1, 1) of -inf}, TensorConstant{(1, 1) of -inf}), Elemwise{Composite{Switch(i0, Switch(i1, i2, i3), i4)}}(Elemwise{Composite{Cast{int8}((GE(i0, i1) * LE(i0, i2)))}}.0, InplaceDimShuffle{0,x}.0, TensorConstant{(1, 1) of ..4393873245}, TensorConstant{(1, 1) of ..2804325936}, TensorConstant{(1, 1) of -inf})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 114, in <module>
    res = pymc3.sample(n)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/sampling.py", line 432, in sample
    trace = _mp_sample(**sample_args)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/sampling.py", line 965, in _mp_sample
    for draw in sampler:
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 393, in __iter__
    draw = ProcessAdapter.recv_draw(self._active)
  File "/usr/local/lib/python3.5/dist-packages/pymc3/parallel_sampling.py", line 297, in recv_draw
    raise error from old_error
RuntimeError: Chain 1 failed.
13

Hmm, could I ask what the values of on_count and schema_count were?

14 
on_count = 3
schema_count = 10

and I represented them as floats too:

on_count = 3.
schema_count = 10.

but I got the same result.

And for completeness:

pTri_given_on = 1.
pTri_given_not_on = .7
tri_delta_on = pTri_given_on - pTri_given_not_on
15

Yes, we need to write a notebook on shape handling.

Your confusion is really common. There are two quantities that characterize a mixture: the number of components (distributions that go into the mixture, in your case 2), and the shape of the outputed value (what the shape of a single call to random() is). The mixtures’ components can be specified in one of two ways:

  1. As a list of distributions. The len of the list specifies the number of components
  2. As a single multidimensional distribution. The shape[-1] of this distribution encodes the number of components.

So what’s the mixtures’ shape? Well, if your components are scalars, like yours, your mixtures’ distribution shape should also be 1 (or left blank which is basically the same). Why did shape=2 work? Because it assumed that you were stacking 2 independent scalars that came from the same underlying mixture distribution, much like what is assumed when you do Normal(..., shape=2).

In the first case, the len of the list informs the number of mixture components
has the number of componente

1 Like
16

Thanks, I’ll try to reproduce your error tomorrow and get back to you

1 Like
17

Ah, ok this makes a lot of sense. Thank you!

18

I don’t mean to rush you, but have you had any luck reproducing the error?

19

I managed to reproduce your error. To fix it you just have to declare the mixtures dtype as int64:

n = 1000
on_count = 3
schema_count = 10
pTri_given_on = 1.
pTri_given_not_on = .7
tri_delta_on = pTri_given_on - pTri_given_not_on
with pymc3.Model() as model:
    # On
    pOn = pymc3.Beta('pOn', alpha=on_count, beta=(schema_count - on_count))
    on = pymc3.Bernoulli('on', p=pOn)

    # Triangle
    triangle_mixture_weights = [on, (1. - on)]
    tri_giv_on = pymc3.Bernoulli.dist(pTri_given_not_on + tri_delta_on)
    tri_giv_not_on = pymc3.Bernoulli.dist(pTri_given_not_on)
    triangle = pymc3.Mixture('triangle', w=triangle_mixture_weights,
                             comp_dists=[tri_giv_on, tri_giv_not_on],
                             shape=2, dtype="int64")
    res = pymc3.sample(n)

Note that you havent passed observed to triangle. When I did, everything worked regardless of passing dtype or not.

1 Like
20

Ah ok, thank you! I’m glad it was a simple fix. Also, shape=1, right?