Collecting ELBO from ADVI

Questions
1

When using new ADVI interface (that uses OPVI framework), is there a way to get the ELBO for each iteration (as we could do with old ADVI interface) ?

What is the average loss computed during the inference?

2

That’s the .hist in the approximated output, see:
http://docs.pymc.io/notebooks/variational_api_quickstart.html#Tracking-parameters

1 Like
3

Thanks @junpenglao.

Can you please tell me that how to access the training time of the models? (in both ADVI or MCMC cases)

or is it accurate to measure the time as shown below?

start = time.time()
apprx = pm.fit(n_iter) #trace = pm.sample(n_draws)
time_take = time.time() - start
4

Isnt the progressbar displays the training/sampling time already?

5

yup it does. Assume if I want to persist those statistics (training time etc) for multiple models. So I was trying to write a script for it.

6

I see, what you did is fine, but just fyi the wall time comparison is usually not very fair, as you usually get a better approximation in MCMC compare to VI.

7

Yeah that is true.

I was thinking to plot a graph: some convergence measure (waic) vs time, and therefore we could understand how each converge with respect to time. That way at least we can understand how each of them behave with respect to time taken for training.

Do you have any suggestions to make it a fair comparison?

1 Like
8

That’s a good idea. Usually for similar comparison, you simulate from a model with known parameters; and then compare the estimations from different methods to this true parameter using some kind of distance/performance measure.

1 Like
9

That is a great idea.

We can quantify the convergence using such technique very easily.

Thanks for the help @junpenglao

10

See also https://arxiv.org/abs/1802.02538 for the difficulty in assessing fit of variational models!

1 Like
11

Thanks @colcarroll. That is an interesting paper. @junpenglao also recommended me this paper in another thread.

When will we see the PSIS and VSBC in PyMC3. FYI, the python scripts for PSIS is already available (done by the authors of the paper)

12

We already have PSIS implemented in PyMC3:

It is, however, an implicit function and not usually expose to users directly.

As for VSBC, once the sample prior function is merged it will be quite straightforward to implement.

13

:grimacing: (have a few projects ahead of that in line for the next few weeks!)

14

No pressure :smile:

1 Like
15

@junpenglao that is good news, did not know about that.