Slicing variables from minibatches

Questions
#1

I’m trying to implement some matrix factorization models for recommender systems where usually there is a sparse matrix X that need to be factorized by the (matrix) product two lower dimensional matrices, say A and B, where the full product of A and B is too large to fit in memory and only want to extract the indices that match to the sparse matrix X.
I was wondering what’s the efficient approach to do this when passing in minibatches of the matrix X as triplets (i, j, value) , i.e. an approach that avoids putting the whole product of the matrices in memory, and ideally also avoids creating a computational graph too large to fit in memory.
Here’s a simplified version of the model I’m trying to build:

# X is some pandas dataframe with columns (row_ind, col_ind, val)
Xbatch=pm.Minibatch(X, batch_size)

with pm.Model():
    A=pm.Normal('A',mu=0,sd=.5, shape=(nrows, latent_dim), testval=Ainit)
    B=pm.Normal('B',mu=0,sd=.5, shape=(ncols, latent_dim), testval=Binit)
    Xhat= ???
    Xvar = pm.Normal('Xvar',mu=Xhat, sd=1.0, observed=Xbatch[:,2])
    approx = pm.fit(method='advi')

If I try:

Xhat = theano.tensor.dot(A, B.T)[(Xbatch[:,0], Xbatch[:,1)]

It tries to compute the whole matrix (thus doesn’t fit in memory), whereas if I try:

Xrow_ind = pm.Minibatch(X.row_ind)
Xcol_ind = pm.Minibatch(X.col_ind)
Xval = pm.Minibatch(X.val)

...
Xhat = theano.tensor.dot(A[Xrow_ind], B[Xcol_ind].T)
...

Then it throws me:
AdvancedSubtensor1.grad illegally returned an integer-valued variable

Whereas if I try it like:

    Xhat=theano.tensor.dot(A.compress(Xrow_ind, axis=0),
                           B.compress(Xcol_ind, axis=0).T)

I get an ‘Index out of bounds’ error.

What would be the proper way to do it?

#2

What you have try seems right and I am not sure why it wont work in your case, for example, the following runs without problem:

X = np.random.randint(1000,size=(10000, 2))
Xval = np.random.randn(10000)
Xbatch = pm.Minibatch(X)
Xval_b = pm.Minibatch(Xval)
nrows = ncols = 1000
latent_dim = 5
with pm.Model():
    A=pm.Normal('A',mu=0,sd=.5, shape=(nrows, latent_dim))
    B=pm.Normal('B',mu=0,sd=.5, shape=(latent_dim, ncols))
    Xhat= pm.math.dot(A[Xbatch[:,0]], B[:,Xbatch[:,1]])
    Xvar = pm.Normal('Xvar',mu=Xhat, sd=1.0, observed=Xval_b)
    approx = pm.fit(method='advi')

Maybe the problem is using pandas dataframe/series in Xrow_ind = pm.Minibatch(X.row_ind). Try converting them into numpy array.

#3

Thanks. That worked in a linux installation, but in windows keep getting errors from theano trying to call blas and failing when trying to use openblas:
error: ('The following error happened while compiling the node', forall_inplace,cpu,scan_fn}(TensorConstant{1}, Elemwise{Composite{((i0 * i1) + i2)}}.0, IncSubtensor{InplaceSet;:int64:}.0, IncSubtensor{InplaceSet;:int64:}.0, TensorConstant{1}, <TensorType(float32, vector)>, <TensorType(int32, matrix)>, Elemwise{Composite{(i0 + Cast{float32}(i1))}}.0, Elemwise{Composite{(i0 + Cast{float32}(i1))}}.0), '\n', 'The following error happened while compiling the node', Dot22(AdvancedSubtensor1.0, InplaceDimShuffle{1,0}.0), '\n', 'bad escape (end of pattern) at position 0', '[Dot22(<TensorType(float32, matrix)>, <TensorType(float32, matrix)>)]')

Guess that’s a problem with theano though and not with pymc.

#4

Thanks for reporting back. Could you please try:

ridx = pm.tt_rng().uniform(size=(10,), low=0, high=X.shape[0]-1e-10).astype('int64')
Xshare = theano.shared(X)
Xbatch = Xshare[ridx]
...
#5

Sorry, just realized I was not using the latest versions of either pymc3 or theano, the problem got solved after updating them (additionally, I had a mess with some installations coming from conda and some from pip and non-official python wheels, perhaps that was causing some trouble).

However, if I try this trick of creating minibatches with a slicing random variable, I get an error:
AdvancedSubtensor1.0 has no test value
For the line that attempts the dot product.

Also, I realized that in the end the problem was in that I was trying to pass in a pandas series with integer type to the observed variable in the pymc3 model, and converting it to float solves the problem in the initial code snippet I posted, e.g. Xval = pm.Minibatch(X.val.astype('float32'))