PyMC3 on Databricks: progress bar

Questions
1

Hello everybody…
I am not sure if this is more of a question for the Databricks community or the PyMC3 one, but anyway… I’ve just tried to run my first pymc3 model on a Databricks notebook, and the pm.sample command does not show the progress bar… can anyone tell me if there is a way to show it?

image
image1367×305 17.4 KB

Thank you!

2

That’s interesting, and I’d guess better for the databricks community.

PyMC3 uses the fastprogress progressbar (link) – if you can’t run their examples in the notebook then it is definitely better for the databricks community!

3

@evivo did you find a solution to this?

4

Unfortunately, no. I have checked that the fastprogress progressbar does not work either and I have posted a message in the databricks forum, no answer so far :frowning:

5

Hello again,

since it does not seem that I will get an answer anytime soon, I would like to ask: is there, or would it be possible to have in a future release, an alternative way to visualize the progress?

For instance, by printing a message every n seconds or something like that…

Thank you

6

Hi,

Could you share a small reproducable sample of codes?

I can then run that codes with the latest PyMC version on databrick and get back to you.

7

Thank you @DanhPhan …

Any code that defines a model and calls pm.sample() would do, for instance the one from this gist

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(42)

true_m = 0.5
true_b = -1.3
true_σ = 0.3

x = np.sort(np.random.uniform(0, 5, 50))
y_true = true_b + true_m * x
y = y_true + true_σ * np.random.randn(len(x))


import pymc3 as pm

with pm.Model() as model:
    # Define the priors on each parameter.
    m = pm.Uniform("m", lower=-5, upper=5)
    b = pm.Uniform("b", lower=-5, upper=5)
    σ = pm.Uniform("σ", lower=0, upper=10)

    # Define the likelihood.
    pm.Normal("obs", mu=m * x + b, sd=σ, observed=y)

    # This is how you will sample the model.
    trace = pm.sample(draws=1000, tune=1000, chains=8, cores=24)

Thank you!

1 Like
8

Hi @evivo

I have run the codes in Databricks, and the results are similar like what you showed. It only show “running…” and the message at the end, without any progress bar.

This is the result using pymc3 version 3.11.4

Capture1
Capture11912×849 123 KB

And this is the result using pymc v4 version 4.0.0b6

Capture2
Capture21917×861 100 KB

However, when I run this notebook in Microsoft Azure notebook, it works. We can see the progress bar as follow:

Capture3
Capture31903×731 110 KB

So, it seems to me that the progress bar is not working on Databricks notebook yet.