Minibatch variables not behaving as expected

Questions
1

I’m having some issues using minibatch tensors when sampling from ADVI with my model.

Creating the minibatch variables and checking the test point results in the test point for the likelihood evaluating to infinity. After some digging, I think it’s because of the eval() that’s happening on the minibatch which is causing the indexes to misalign. I have the minimal code needed to reproduce this in the gist below.

Further, using shared variable alone seems to be doing the correct thing.

I was wondering if there is something I’m doing wrong when specifying the model with the minibatch variables.

Any help would be appreciated.

Here is some test data I’ve been using:

anonymized_data.csv (2.5 MB)

2

Thanks for reporting, I think this is a bug. Even specifying the random_seed in pm.Minibatch doesnt help. cc @ferrine

Before we fix it, try below instead:

import theano
batch_size = 2000

random_seed = 42

ridx = pm.tt_rng().uniform(size=(batch_size,),
                           low=0, high=train.y.values.shape[0]-1e-10).astype('int64')

model_variables = {}

# Nominal Predictors of model
for predictor in encoders:
    encoder = encoders[predictor]
    model_variables[predictor] = theano.shared(train[predictor].values)[ridx]

model_variables['y_obs'] = theano.shared(train.y.values)[ridx]
model_variables['n'] = theano.shared(train.n.values)[ridx]

model = construct_model(model_variables, encoders)
3

Oh looking at your model I found out why: you evaluated one of the shared value already I think that makes the shared minibatch out of sync:

shape=model_variables['y_obs'].eval().size,

change your model definition to accept a shape like def construct_model(model_variables, encoders, muBeta_shape) should fix your problem.

4

Ok i’ll test that out.

So to be clear, muBeta_shape should be the batch_size right?

5

exactly, and if you are passing all the data then it is train.y.values.shape

6

Hi @junpenglao. This solved my initial problem. However, I’m still having trouble training the model using ADVI and minibatch and then drawing from ppc with the test values.

I’m trying to follow along with the suggestions here: PPC with Minibatch ADVI, but I get a dimension input mismatch error:

ValueError: Input dimension mis-match. (input[0].shape[0] = 85054, input[3].shape[0] = 2000)

How can I resolve this error?

7

For minibatch to work with sample_ppc you need to change your way to model. More details here: https://github.com/pymc-devs/pymc3/issues/2190#issuecomment-311609342

8

Hi @junpenglao, that’s precisely what I’m doing:

# Create shared variables. Use these to initialize model
model_variables = {}

# Nominal Predictors of model
for predictor in encoders:
    encoder = encoders[predictor]
    model_variables[predictor] = shared(encoder.transform(train[predictor]))

model_variables['y_obs'] = shared(train.y.values)
model_variables['n'] = shared(train.n.values)

# Create minibatch variables
batch_size = 2000
random_seed = 42
minibatch_variables = {}

# Nominal Predictors of model
for predictor in encoders:
    encoder = encoders[predictor]
    minibatch_variables[predictor] = pm.Minibatch(
        encoder.transform(train[predictor]), batch_size=batch_size
    )

minibatch_variables['y_obs'] = pm.Minibatch(train.y.values, batch_size=batch_size)
minibatch_variables['n'] = pm.Minibatch(train.n.values, batch_size=batch_size)

# Replacements to be used for training
minibatch_replacements = {model_variables[key]: minibatch_variables[key] for key in model_variables} 

model = construct_model(model_variables, encoders, train.y.values.size)

with model:    
    n = 5000
    advifit = pm.fit(
        n, callbacks=[pm.callbacks.CheckParametersConvergence(tolerance=1e-4)],
        more_replacements=minibatch_replacements
    )

which is giving the following TraceBack:

ValueError Traceback (most recent call last)
/anaconda/envs/numerical/lib/python3.6/site-packages/theano/compile/function_module.py in call(self, *args, **kwargs)
883 outputs =
–> 884 self.fn() if output_subset is None else
885 self.fn(output_subset=output_subset)

ValueError: Input dimension mis-match. (input[0].shape[0] = 85054, input[3].shape[0] = 2000)

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
in ()
3 advifit = pm.fit(
4 n, callbacks=[pm.callbacks.CheckParametersConvergence(tolerance=1e-4)],
----> 5 more_replacements=minibatch_replacements
6 )

/anaconda/envs/numerical/lib/python3.6/site-packages/pymc3/variational/inference.py in fit(n, local_rv, method, model, random_seed, start, inf_kwargs, **kwargs)
754 ‘or Inference instance’ %
755 set(_select.keys()))
–> 756 return inference.fit(n, **kwargs)

/anaconda/envs/numerical/lib/python3.6/site-packages/pymc3/variational/inference.py in fit(self, n, score, callbacks, progressbar, **kwargs)
133 with tqdm.trange(n, disable=not progressbar) as progress:
134 if score:
–> 135 state = self._iterate_with_loss(0, n, step_func, progress, callbacks)
136 else:
137 state = self._iterate_without_loss(0, n, step_func, progress, callbacks)

/anaconda/envs/numerical/lib/python3.6/site-packages/pymc3/variational/inference.py in _iterate_with_loss(self, s, n, step_func, progress, callbacks)
175 try:
176 for i in progress:
–> 177 e = step_func()
178 if np.isnan(e): # pragma: no cover
179 scores = scores[:i]

/anaconda/envs/numerical/lib/python3.6/site-packages/theano/compile/function_module.py in call(self, *args, **kwargs)
896 node=self.fn.nodes[self.fn.position_of_error],
897 thunk=thunk,
–> 898 storage_map=getattr(self.fn, ‘storage_map’, None))
899 else:
900 # old-style linkers raise their own exceptions

/anaconda/envs/numerical/lib/python3.6/site-packages/theano/gof/link.py in raise_with_op(node, thunk, exc_info, storage_map)
323 # extra long error message in that case.
324 pass
–> 325 reraise(exc_type, exc_value, exc_trace)
326
327

/anaconda/envs/numerical/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
690 value = tp()
691 if value.traceback is not tb:
–> 692 raise value.with_traceback(tb)
693 raise value
694 finally:

/anaconda/envs/numerical/lib/python3.6/site-packages/theano/compile/function_module.py in call(self, *args, **kwargs)
882 try:
883 outputs =
–> 884 self.fn() if output_subset is None else
885 self.fn(output_subset=output_subset)
886 except Exception:

ValueError: Input dimension mis-match. (input[0].shape[0] = 85054, input[3].shape[0] = 2000)

9

But you can not create a model like this, because in one of the nodes you specified the shape to have shape=train.y.values.size, so now when you are trying to replace it with a minibatch of shape 2000 it is not going to work.

Try to change the way you write down the model so that you dont need to specify the shape for that node mu. Otherwise, maybe it is easier to cherry pick the code from pm.sample_ppc to a separate function for ppc sampling.

10

For the hierarchical logistic regression model, how can it be rewritten to not include that shape parameter?

11

Try using betabinomial distribution directly http://docs.pymc.io/api/distributions/discrete.html#pymc3.distributions.discrete.BetaBinomial

12

I’ll give this a shot. Thank you.