Collecting ELBO from ADVI

When using new ADVI interface (that uses OPVI framework), is there a way to get the ELBO for each iteration (as we could do with old ADVI interface) ?

What is the average loss computed during the inference?

That’s the .hist in the approximated output, see:

1 Like

Thanks @junpenglao.

Can you please tell me that how to access the training time of the models? (in both ADVI or MCMC cases)

or is it accurate to measure the time as shown below?

start = time.time()
apprx = #trace = pm.sample(n_draws)
time_take = time.time() - start

Isnt the progressbar displays the training/sampling time already?

yup it does. Assume if I want to persist those statistics (training time etc) for multiple models. So I was trying to write a script for it.

I see, what you did is fine, but just fyi the wall time comparison is usually not very fair, as you usually get a better approximation in MCMC compare to VI.

Yeah that is true.

I was thinking to plot a graph: some convergence measure (waic) vs time, and therefore we could understand how each converge with respect to time. That way at least we can understand how each of them behave with respect to time taken for training.

Do you have any suggestions to make it a fair comparison?

1 Like

That’s a good idea. Usually for similar comparison, you simulate from a model with known parameters; and then compare the estimations from different methods to this true parameter using some kind of distance/performance measure.

1 Like

That is a great idea.

We can quantify the convergence using such technique very easily.

Thanks for the help @junpenglao

See also for the difficulty in assessing fit of variational models!

1 Like

Thanks @colcarroll. That is an interesting paper. @junpenglao also recommended me this paper in another thread.

When will we see the PSIS and VSBC in PyMC3. FYI, the python scripts for PSIS is already available (done by the authors of the paper)

We already have PSIS implemented in PyMC3:

It is, however, an implicit function and not usually expose to users directly.

As for VSBC, once the sample prior function is merged it will be quite straightforward to implement.

:grimacing: (have a few projects ahead of that in line for the next few weeks!)

No pressure :smile:

1 Like

@junpenglao that is good news, did not know about that.