# Slicing variables from minibatches

**URL:** <https://discourse.pymc.io/t/slicing-variables-from-minibatches/646>\
**Category:** Questions\
**Created:** [December 12, 2017, 10:04pm UTC](https://discourse.pymc.io/t/slicing-variables-from-minibatches/646 "2017-12-12T22:04:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![david.cortes](https://avatars.discourse-cdn.com/v4/letter/d/90db22/32.png) [@david.cortes](https://discourse.pymc.io/u/david.cortes)\
**Post date:** [December 12, 2017, 10:04pm UTC](https://discourse.pymc.io/t/slicing-variables-from-minibatches/646/1 "2017-12-12T22:04:35Z")

</div>

I’m trying to implement some matrix factorization models for recommender systems where usually there is a sparse matrix X that need to be factorized by the (matrix) product two lower dimensional matrices, say A and B, where the full product of A and B is too large to fit in memory and only want to extract the indices that match to the sparse matrix X.  
I was wondering what’s the efficient approach to do this when passing in minibatches of the matrix X as triplets (i, j, value) , i.e. an approach that avoids putting the whole product of the matrices in memory, and ideally also avoids creating a computational graph too large to fit in memory.  
Here’s a simplified version of the model I’m trying to build:

```
# X is some pandas dataframe with columns (row_ind, col_ind, val)
Xbatch=pm.Minibatch(X, batch_size)

with pm.Model():
    A=pm.Normal('A',mu=0,sd=.5, shape=(nrows, latent_dim), testval=Ainit)
    B=pm.Normal('B',mu=0,sd=.5, shape=(ncols, latent_dim), testval=Binit)
    Xhat= ???
    Xvar = pm.Normal('Xvar',mu=Xhat, sd=1.0, observed=Xbatch[:,2])
    approx = pm.fit(method='advi')

```

If I try:

```
Xhat = theano.tensor.dot(A, B.T)[(Xbatch[:,0], Xbatch[:,1)]

```

It tries to compute the whole matrix (thus doesn’t fit in memory), whereas if I try:

```
Xrow_ind = pm.Minibatch(X.row_ind)
Xcol_ind = pm.Minibatch(X.col_ind)
Xval = pm.Minibatch(X.val)

...
Xhat = theano.tensor.dot(A[Xrow_ind], B[Xcol_ind].T)
...

```

Then it throws me:  
`AdvancedSubtensor1.grad illegally returned an integer-valued variable`

Whereas if I try it like:

```auto
    Xhat=theano.tensor.dot(A.compress(Xrow_ind, axis=0),
                           B.compress(Xcol_ind, axis=0).T)

```

I get an ‘Index out of bounds’ error.

What would be the proper way to do it?

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [December 14, 2017, 9:40am UTC](https://discourse.pymc.io/t/slicing-variables-from-minibatches/646/2 "2017-12-14T09:40:29Z")

</div>

What you have try seems right and I am not sure why it wont work in your case, for example, the following runs without problem:

```python
X = np.random.randint(1000,size=(10000, 2))
Xval = np.random.randn(10000)
Xbatch = pm.Minibatch(X)
Xval_b = pm.Minibatch(Xval)
nrows = ncols = 1000
latent_dim = 5
with pm.Model():
    A=pm.Normal('A',mu=0,sd=.5, shape=(nrows, latent_dim))
    B=pm.Normal('B',mu=0,sd=.5, shape=(latent_dim, ncols))
    Xhat= pm.math.dot(A[Xbatch[:,0]], B[:,Xbatch[:,1]])
    Xvar = pm.Normal('Xvar',mu=Xhat, sd=1.0, observed=Xval_b)
    approx = pm.fit(method='advi')

```

Maybe the problem is using pandas dataframe/series in `Xrow_ind = pm.Minibatch(X.row_ind)`. Try converting them into numpy array.

---

<div class="post-metadata">

**Author:** ![david.cortes](https://avatars.discourse-cdn.com/v4/letter/d/90db22/32.png) [@david.cortes](https://discourse.pymc.io/u/david.cortes)\
**Post date:** [December 15, 2017, 6:20pm UTC](https://discourse.pymc.io/t/slicing-variables-from-minibatches/646/3 "2017-12-15T18:20:51Z")

</div>

Thanks. That worked in a linux installation, but in windows keep getting errors from theano trying to call blas and failing when trying to use openblas:  
`error: ('The following error happened while compiling the node', forall_inplace,cpu,scan_fn}(TensorConstant{1}, Elemwise{Composite{((i0 * i1) + i2)}}.0, IncSubtensor{InplaceSet;:int64:}.0, IncSubtensor{InplaceSet;:int64:}.0, TensorConstant{1}, <TensorType(float32, vector)>, <TensorType(int32, matrix)>, Elemwise{Composite{(i0 + Cast{float32}(i1))}}.0, Elemwise{Composite{(i0 + Cast{float32}(i1))}}.0), '\n', 'The following error happened while compiling the node', Dot22(AdvancedSubtensor1.0, InplaceDimShuffle{1,0}.0), '\n', 'bad escape (end of pattern) at position 0', '[Dot22(<TensorType(float32, matrix)>, <TensorType(float32, matrix)>)]')`

Guess that’s a problem with theano though and not with pymc.

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [December 15, 2017, 6:47pm UTC](https://discourse.pymc.io/t/slicing-variables-from-minibatches/646/4 "2017-12-15T18:47:06Z")

</div>

Thanks for reporting back. Could you please try:

```python
ridx = pm.tt_rng().uniform(size=(10,), low=0, high=X.shape[0]-1e-10).astype('int64')
Xshare = theano.shared(X)
Xbatch = Xshare[ridx]
...

```

---

<div class="post-metadata">

**Author:** ![david.cortes](https://avatars.discourse-cdn.com/v4/letter/d/90db22/32.png) [@david.cortes](https://discourse.pymc.io/u/david.cortes)\
**Post date:** [December 16, 2017, 12:17pm UTC](https://discourse.pymc.io/t/slicing-variables-from-minibatches/646/5 "2017-12-16T12:17:42Z")

</div>

Sorry, just realized I was not using the latest versions of either pymc3 or theano, the problem got solved after updating them (additionally, I had a mess with some installations coming from conda and some from pip and non-official python wheels, perhaps that was causing some trouble).

However, if I try this trick of creating minibatches with a slicing random variable, I get an error:  
`AdvancedSubtensor1.0 has no test value`  
For the line that attempts the dot product.

Also, I realized that in the end the problem was in that I was trying to pass in a pandas series with integer type to the observed variable in the pymc3 model, and converting it to float solves the problem in the initial code snippet I posted, e.g. `Xval = pm.Minibatch(X.val.astype('float32'))`