# Minibatch for a large dataset ADVI

**URL:** <https://discourse.pymc.io/t/minibatch-for-a-large-dataset-advi/1867>\
**Category:** Questions\
**Created:** [September 5, 2018, 2:49pm UTC](https://discourse.pymc.io/t/minibatch-for-a-large-dataset-advi/1867 "2018-09-05T14:49:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gaddamanil16](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/gaddamanil16/32/927_2.png) [@gaddamanil16](https://discourse.pymc.io/u/gaddamanil16)\
**Post date:** [September 5, 2018, 2:49pm UTC](https://discourse.pymc.io/t/minibatch-for-a-large-dataset-advi/1867/1 "2018-09-05T14:49:57Z")

</div>

Hi,

I am dealing with a very large dataset and trying to run Latent Dirichlet Allocation using ADVI on it. I used scikit learn `CountVectorizer` to prepare the data and the resultant sparse matrix,which is seen as observed data in the likelihood. But I can’t use `toarray` to get the dense version of the sparse matrix because it throws memory error. So I tried to use a for loop so that I can run same model for parts of data. But the problem arises while sampling. How do i combine all the traces, but before that how to save all the trace objects.

```
'''
For minibatch 
'''
partitions = 25
index_number = 0
splits = int(tf26.shape[0]/partitions)
#tr_coll = np.array((25))
tr_coll = []
tf26 = tf26

for i in range(partitions):
    indices = np.arange(index_number,index_number+splits)
    data = tf26[indices,:].toarray()
    
    (D, W) = data.shape #In this case I see V and W of same length  
    
    LDA_output = theano.shared(data) #Throws error because data is a sparse matrix 
    minibatch_data = pm.Minibatch(data, batch_size=1000)
    
    with model: 
        theta = pm.Dirichlet("thetas_%d" %i, a=alpha, shape=(D, K))
        phi = pm.Dirichlet("phis_%d" %i, a=beta, shape=(K, V))
        doc = pm.DensityDist("doc_%d" %i, log_lda(theta,phi), observed=minibatch_data)   
    with model:    
        inference = pm.ADVI()
        approx = pm.fit(n=1000,method=inference,more_replacements = {LDA_output:minibatch_data})
    tr = approx.sample(draws=500)
    
    index_number = index_number+splits+1
    
    tr_coll[i] = tr

```

Any suggestions could be of great help!  
Thanks

---

<div class="post-metadata">

**Author:** ![junpenglao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/junpenglao/32/8_2.png) [@junpenglao](https://discourse.pymc.io/u/junpenglao)\
**Post date:** [September 7, 2018, 9:20am UTC](https://discourse.pymc.io/t/minibatch-for-a-large-dataset-advi/1867/2 "2018-09-07T09:20:23Z")

</div>

Try saving the traces using using `pm.save_trace` and `pm.load_trace`. See also [Saving and Loading GP model in PYMC3](https://discourse.pymc.io/t/saving-and-loading-gp-model-in-pymc3/1801/2?u=junpenglao)

---

<div class="post-metadata">

**Author:** ![gaddamanil16](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/gaddamanil16/32/927_2.png) [@gaddamanil16](https://discourse.pymc.io/u/gaddamanil16)\
**Post date:** [September 7, 2018, 7:39pm UTC](https://discourse.pymc.io/t/minibatch-for-a-large-dataset-advi/1867/3 "2018-09-07T19:39:03Z")

</div>

The problem in my case is that rerunning the model is taking a very long time. So if I use `pm.save_trace` I would have to do that. And that is not quite feasible for me because of my constraints.