What are the differences between NUTS and ADVI?

Hi! I’m new in the Bayesian / PyMC3 world, and I’ve come across a lot the NUTS and ADVI methods for doing inference. My understanding is that you should avoid using the find_MAP function, and instead use one of these two. Can anyone provide any overall summary of the difference between them? Is it fair to say that ADVI is faster (but less accurate) than NUTS? Any help (and useful references) is highly appreciated!

Adding my two cents having used the PyMC3 implementations of both on a range of nonstandard problems:

  1. When it is said that ADVI is “less accurate”, it is often made as a general comment about the inability of all variational inference methods to properly characterize posterior variance. For example, the ADVI posterior for a regression coefficient will often be too narrow / or concentrated. This is because of the fact that VI methods are minimizing a loss function which asymmetrically favors using too-narrow approximations.

  2. It has often occurred to me when working with weird models with very difficult posterior geometries such that all MCMC methods fail, initializing first with ADVI can help get the samplers to work. I suspect that this is because the initial values for the MCMC sampler may be poorly chosen, leading to horrible numerical issues for Hamiltonian Monte Carlo; it appears that in many cases ADVI is somewhat more robust.

  3. Some problems are just too big for NUTS (even with a GPU) and ADVI is the only option for model fitting. I’ve used ADVI + GPU to train deep convolutional autoencoders with 10 million+ parameters and Bayesian regularization using the minibatched implementation of ADVI within PyMC3.

As a short summary, I think the worst use case for ADVI would be with a small dataset and complicated model structure while it is best used for very large models and datasets requiring minibatched computation.

5 Likes

Hi @ckrapu ,

Is there a way to do this ADVI initialization in PyMC?

Can you elaborate more on this?

Hey @jroberayalas ,

  1. Initializing with ADVI is really easy. By default, pm.sample calls a NUTSampler
    trace = pm.sample(10000, init="advi")
    pymc.init_nuts — PyMC 5.5.0 documentation

  2. ADVI works by calculating the gradient of the ELBO at each point on its path. The ELBO is ideally computed across all samples, but in the case of batching, only a subsample is used to approximate the ELBO. This makes it able to handle more datapoints with less precision, which the creators think to be a worthwhile tradeoff (see section 4.3)
    https://arxiv.org/pdf/1603.00788.pdf

1 Like