ADVI in PyMC3 3.2 is approximately two times slower than 3.1

Hi all,

I wonder whether anyone else has noticed this issue: in my case, ADVI in PyMC3 v3.2 is approximately two times slower (theano 0.9.0 and everything identical except for the pymc3 version). Also, compilation time for the step function is ~ 3x longer and uses 5x more memory which unfortunately in my case is a deal breaker and I need to stick to 3.1. I wonder whether this has something to do with the major upgrade to OPVI and grouped approximation in 3.2?

ps> the “legacy” ADVI implementation (variational.advi) has the same decent performance in 3.1 and 3.2, which leads me to believe that the performance regression has something to do with the OPVI framework.

Best,
Mehrtash

Hi Mehrtash,

The legacy ADVI have not changed between 3.1 and 3.2, so I wouldn’t expect any change of performance.

As for the slowdown of ADVI implementation between 3.1 and 3.2, it is quite likely as the new framework is more flexible but also computationally more intensive. This is related to some of the theano bottleneck @ferrine faces when he is implementing the group approximation, so it is likely quite difficult to improve from our end.

what do you mean by step function?

Hi, new ADVI is my work. I suppose that step function is the main training function used for train iterations. OPVI has been redesigned several times. I think that the source of the problem is a lot of theano.clone operations, but without them It was not possible to make new things work.

Hi Maxim and Junpeng,

Thanks for your replies. I need to study the new opvi further (it is quite complex). I noticed the numerous appearances of theano.clone indeed during my first reading.

Best,
Mehrtash

On weekends I tried to make it work. The problem is that theano clone fails to replace ScanOp when it is not needed (https://github.com/pymc-devs/pymc3/blob/master/pymc3/variational/opvi.py#L1379)
I failed to find the source of the problem quickly

Solved by https://github.com/pymc-devs/pymc3/pull/2759