DEMetropolis stalls with progressbar=True

Questions v5
,
1

Hi everyone,

I rely on MCMC progress and estimated remaining time to schedule subsequent computations. With parallel DEMetropolis, enabling the progress bar appears to stall my job. Disabling it allows completion, but leaves me without the progress/ETA information I need. The DEMetropolisZ sampler, on the other hand, works with the progressbar but its convergence performance is not good. Is there a way to retain a functioning progress display for DEMetropolis?

Matched in-memory controls

I ran two standalone Python scripts on the same Linux cluster environment. They differ only in progressbar=True versus False. Both use the default in-memory trace, the same model, observations, 24 deterministic starts, seed inputs, and 100 tuning plus 100 retained iterations per chain.

Case Chains / cores progressbar Result
A 24 / 24 False Completes sampling in 211 seconds; posterior dimensions are 24 chains by 100 draws
B 24 / 24 True Appears to stall for more than 12 hours with extremely low CPU utilization

The sampling call is:

idata = pm.sample(
    100, tune=100, chains=24, cores=24,
    initvals=initvals, random_seed=chain_seeds,
    discard_tuned_samples=False,
    step=ScalarStatsDEMetropolis(tune="scaling"),
    progressbar=True,  # Changing only this to False completes.
)

More details can be checked out in the attached python script (I have obsecured the forward model parts as they involve unpublished work) and run outputs
run_outputs.txt (6.9 KB)
test_mcmc_script.py (6.6 KB)

A progress/console locking interaction is one hypothesis; I have not yet collected parent or worker stack dumps.

Related versions of PyMC environment

Python: 3.10.19
pymc: 5.25.1
pytensor: 2.31.7
arviz: 0.23.1
numpy: 2.2.6
scipy: 1.15.2

2

Have you run this on jupyter? If so did you try direct terminal script?

3

Hi Ricardo, I’m running python scripts directly from the terminal inside a Docker container on a Linux cluster, not through Jupyter.

4

reproduced it, some issue with rich/multiprocessing ordering… Will see if we can do it like pm.sample does

5

Perhaps fixed by Fix hang in DEMetropolis due to progressbar by ricardoV94 · Pull Request #8421 · pymc-devs/pymc · GitHub

6

Can you try Release v6.3.2 · pymc-devs/pymc · GitHub ?

7

I have not tried the latest release, but I did apply the fix by you (#8421) to my current pymc environment and it works well!