# Minibatch variables not behaving as expected

**URL:** <https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927>\
**Category:** Questions\
**Created:** [March 12, 2018, 2:33pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927 "2018-03-12T14:33:32Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![researcher](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@researcher](https://discourse.pymc.io/u/researcher)\
**Post date:** [March 12, 2018, 2:33pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/1 "2018-03-12T14:33:32Z")

</div>

I’m having some issues using minibatch tensors when sampling from ADVI with my model.

Creating the minibatch variables and checking the test point results in the test point for the likelihood evaluating to infinity. After some digging, I think it’s because of the eval() that’s happening on the minibatch which is causing the indexes to misalign. I have the minimal code needed to reproduce this in the gist below.

Further, using shared variable alone seems to be doing the correct thing.

> <https://gist.github.com/anonymous/50c8dc69fb9dea41eea5032cf5e31cd7>

I was wondering if there is something I’m doing wrong when specifying the model with the minibatch variables.

Any help would be appreciated.

Here is some test data I’ve been using:

[anonymized\_data.csv](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/1X/b15d759d476772deafdcef29045156c4d5994777.csv) (2.5 MB)

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 12, 2018, 4:35pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/2 "2018-03-12T16:35:07Z")

</div>

Thanks for reporting, I think this is a bug. Even specifying the `random_seed` in `pm.Minibatch` doesnt help. cc @ferrine

Before we fix it, try below instead:

```python
import theano
batch_size = 2000

random_seed = 42

ridx = pm.tt_rng().uniform(size=(batch_size,),
                           low=0, high=train.y.values.shape[0]-1e-10).astype('int64')

model_variables = {}

# Nominal Predictors of model
for predictor in encoders:
    encoder = encoders[predictor]
    model_variables[predictor] = theano.shared(train[predictor].values)[ridx]

model_variables['y_obs'] = theano.shared(train.y.values)[ridx]
model_variables['n'] = theano.shared(train.n.values)[ridx]

model = construct_model(model_variables, encoders)

```

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 12, 2018, 4:44pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/3 "2018-03-12T16:44:42Z")

</div>

Oh looking at your model I found out why: you evaluated one of the shared value already I think that makes the shared minibatch out of sync:

```python
shape=model_variables['y_obs'].eval().size,

```

change your model definition to accept a `shape` like `def construct_model(model_variables, encoders, muBeta_shape)` should fix your problem.

---

<div class="post-metadata">

**Author:** ![researcher](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@researcher](https://discourse.pymc.io/u/researcher)\
**Post date:** [March 12, 2018, 5:51pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/4 "2018-03-12T17:51:44Z")

</div>

Ok i’ll test that out.

So to be clear, muBeta\_shape should be the batch\_size right?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 12, 2018, 5:56pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/5 "2018-03-12T17:56:56Z")

</div>

exactly, and if you are passing all the data then it is `train.y.values.shape`

---

<div class="post-metadata">

**Author:** ![researcher](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@researcher](https://discourse.pymc.io/u/researcher)\
**Post date:** [March 13, 2018, 8:40pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/6 "2018-03-13T20:40:15Z")

</div>

Hi @junpenglao. This solved my initial problem. However, I’m still having trouble training the model using ADVI and minibatch and then drawing from ppc with the test values.

I’m trying to follow along with the suggestions here: [PPC with Minibatch ADVI](https://discourse.pymc.io/t/ppc-with-minibatch-advi/131), but I get a dimension input mismatch error:

```
ValueError: Input dimension mis-match. (input[0].shape[0] = 85054, input[3].shape[0] = 2000)

```

How can I resolve this error?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 13, 2018, 10:27pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/7 "2018-03-13T22:27:38Z")

</div>

For minibatch to work with sample\_ppc you need to change your way to model. More details here: [https://github.com/pymc-devs/pymc3/issues/2190#issuecomment-311609342](https://github.com/pymc-devs/pymc3/issues/2190#issuecomment-311609342)

---

<div class="post-metadata">

**Author:** ![researcher](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@researcher](https://discourse.pymc.io/u/researcher)\
**Post date:** [March 13, 2018, 10:39pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/8 "2018-03-13T22:39:10Z")

</div>

Hi @junpenglao, that’s precisely what I’m doing:

```
# Create shared variables. Use these to initialize model
model_variables = {}

# Nominal Predictors of model
for predictor in encoders:
    encoder = encoders[predictor]
    model_variables[predictor] = shared(encoder.transform(train[predictor]))

model_variables['y_obs'] = shared(train.y.values)
model_variables['n'] = shared(train.n.values)

# Create minibatch variables
batch_size = 2000
random_seed = 42
minibatch_variables = {}

# Nominal Predictors of model
for predictor in encoders:
    encoder = encoders[predictor]
    minibatch_variables[predictor] = pm.Minibatch(
        encoder.transform(train[predictor]), batch_size=batch_size
    )

minibatch_variables['y_obs'] = pm.Minibatch(train.y.values, batch_size=batch_size)
minibatch_variables['n'] = pm.Minibatch(train.n.values, batch_size=batch_size)

# Replacements to be used for training
minibatch_replacements = {model_variables[key]: minibatch_variables[key] for key in model_variables} 

model = construct_model(model_variables, encoders, train.y.values.size)

with model:    
    n = 5000
    advifit = pm.fit(
        n, callbacks=[pm.callbacks.CheckParametersConvergence(tolerance=1e-4)],
        more_replacements=minibatch_replacements
    )

```

which is giving the following TraceBack:

* * *

ValueError Traceback (most recent call last)  
/anaconda/envs/numerical/lib/python3.6/site-packages/theano/compile/function\_module.py in **call** (self, \*args, \*\*kwargs)  
883 outputs =  
–\> 884 self.fn() if output\_subset is None else  
885 self.fn(output\_subset=output\_subset)

ValueError: Input dimension mis-match. (input[0].shape[0] = 85054, input[3].shape[0] = 2000)

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)  
 in ()  
3 advifit = pm.fit(  
4 n, callbacks=[pm.callbacks.CheckParametersConvergence(tolerance=1e-4)],  
----\> 5 more\_replacements=minibatch\_replacements  
6 )

/anaconda/envs/numerical/lib/python3.6/site-packages/pymc3/variational/inference.py in fit(n, local\_rv, method, model, random\_seed, start, inf\_kwargs, \*\*kwargs)  
754 ‘or Inference instance’ %  
755 set(\_select.keys()))  
–\> 756 return inference.fit(n, \*\*kwargs)

/anaconda/envs/numerical/lib/python3.6/site-packages/pymc3/variational/inference.py in fit(self, n, score, callbacks, progressbar, \*\*kwargs)  
133 with tqdm.trange(n, disable=not progressbar) as progress:  
134 if score:  
–\> 135 state = self.\_iterate\_with\_loss(0, n, step\_func, progress, callbacks)  
136 else:  
137 state = self.\_iterate\_without\_loss(0, n, step\_func, progress, callbacks)

/anaconda/envs/numerical/lib/python3.6/site-packages/pymc3/variational/inference.py in \_iterate\_with\_loss(self, s, n, step\_func, progress, callbacks)  
175 try:  
176 for i in progress:  
–\> 177 e = step\_func()  
178 if np.isnan(e): # pragma: no cover  
179 scores = scores[:i]

/anaconda/envs/numerical/lib/python3.6/site-packages/theano/compile/function\_module.py in **call** (self, \*args, \*\*kwargs)  
896 node=self.fn.nodes[self.fn.position\_of\_error],  
897 thunk=thunk,  
–\> 898 storage\_map=getattr(self.fn, ‘storage\_map’, None))  
899 else:  
900 # old-style linkers raise their own exceptions

/anaconda/envs/numerical/lib/python3.6/site-packages/theano/gof/link.py in raise\_with\_op(node, thunk, exc\_info, storage\_map)  
323 # extra long error message in that case.  
324 pass  
–\> 325 reraise(exc\_type, exc\_value, exc\_trace)  
326  
327

/anaconda/envs/numerical/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)  
690 value = tp()  
691 if value. **traceback** is not tb:  
–\> 692 raise value.with\_traceback(tb)  
693 raise value  
694 finally:

/anaconda/envs/numerical/lib/python3.6/site-packages/theano/compile/function\_module.py in **call** (self, \*args, \*\*kwargs)  
882 try:  
883 outputs =  
–\> 884 self.fn() if output\_subset is None else  
885 self.fn(output\_subset=output\_subset)  
886 except Exception:

ValueError: Input dimension mis-match. (input[0].shape[0] = 85054, input[3].shape[0] = 2000)

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 13, 2018, 10:45pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/9 "2018-03-13T22:45:01Z")

</div>

But you can not create a model like this, because in one of the nodes you specified the shape to have `shape=train.y.values.size`, so now when you are trying to replace it with a minibatch of shape 2000 it is not going to work.

Try to change the way you write down the model so that you dont need to specify the shape for that node `mu`. Otherwise, maybe it is easier to cherry pick the code from pm.sample\_ppc to a separate function for ppc sampling.

---

<div class="post-metadata">

**Author:** ![researcher](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@researcher](https://discourse.pymc.io/u/researcher)\
**Post date:** [March 13, 2018, 10:46pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/10 "2018-03-13T22:46:27Z")

</div>

For the hierarchical logistic regression model, how can it be rewritten to not include that shape parameter?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [March 13, 2018, 10:58pm UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/11 "2018-03-13T22:58:38Z")

</div>

Try using betabinomial distribution directly [http://docs.pymc.io/api/distributions/discrete.html#pymc3.distributions.discrete.BetaBinomial](http://docs.pymc.io/api/distributions/discrete.html#pymc3.distributions.discrete.BetaBinomial)

---

<div class="post-metadata">

**Author:** ![researcher](https://avatars.discourse-cdn.com/v4/letter/r/edb3f5/32.png) [@researcher](https://discourse.pymc.io/u/researcher)\
**Post date:** [March 14, 2018, 1:30am UTC](https://discourse.pymc.io/t/minibatch-variables-not-behaving-as-expected/927/12 "2018-03-14T01:30:16Z")

</div>

I’ll give this a shot. Thank you.