I wonder whether anyone else has noticed this issue: in my case, ADVI in PyMC3 v3.2 is approximately two times slower (theano 0.9.0 and everything identical except for the pymc3 version). Also, compilation time for the step function is ~ 3x longer and uses 5x more memory which unfortunately in my case is a deal breaker and I need to stick to 3.1. I wonder whether this has something to do with the major upgrade to OPVI and grouped approximation in 3.2?
ps> the “legacy” ADVI implementation (variational.advi) has the same decent performance in 3.1 and 3.2, which leads me to believe that the performance regression has something to do with the OPVI framework.
The legacy ADVI have not changed between 3.1 and 3.2, so I wouldn’t expect any change of performance.
As for the slowdown of ADVI implementation between 3.1 and 3.2, it is quite likely as the new framework is more flexible but also computationally more intensive. This is related to some of the theano bottleneck @ferrine faces when he is implementing the group approximation, so it is likely quite difficult to improve from our end.
Hi, new ADVI is my work. I suppose that step function is the main training function used for train iterations. OPVI has been redesigned several times. I think that the source of the problem is a lot of theano.clone operations, but without them It was not possible to make new things work.
Thanks for your replies. I need to study the new opvi further (it is quite complex). I noticed the numerous appearances of theano.clone indeed during my first reading.