Real-time MCMC progress bar rendering in non-TTY (SLURM) environments

Questions v5
1

Dear PyMC/Fastprogress Developers,

I am writing to report an issue regarding the real-time monitoring of MCMC progress when running PyMC models in a non-interactive HPC environment (SLURM batch jobs).

Environment Details

  • Installation: Conda environment (conda-forge channel)
  • Python Version: 3.14
  • Library: PyMC (v.5.27.0) with fastprogress
  • Execution: SLURM cluster using python3 -u to redirect stdout to a .out log file.

The Problem: When executing pm.sample(progressbar=True) within a SLURM job, the progress bar often fails to provide real-time feedback. Instead of a verbose log, it renders as a single “Rich” console box (below image) that only appears in the log file once the sampling is 100% complete.

pm.sample(draws=2000, tune=1000,target_accept=0.8, cores=2, chains=2, progressbar=True, return_inferencedata=True)

image
image710×138 3.02 KB

Current Attempt: I have been trying to force a verbose, line-by-line text output using the following configuration at the top of my scripts:

import logging
import fastprogress
from fastprogress.fastprogress import master_bar, progress_bar

fastprogress.fastprogress.printing = True
master_bar.refresh_speed = 10  

logger = logging.getLogger(“pymc”)
logger.setLevel(logging.INFO)
if not logger.handlers:
    handler = logging.StreamHandler(sys.stdout)
    handler.setLevel(logging.INFO)
    formatter = logging.Formatter(‘%(asctime)s - %(name)s - %(levelname)s - %(message)s’)
    handler.setFormatter(formatter)
    logger.addHandler(handler)

Question: Is there a recommended or “native” way to configure PyMC/Fastprogress so that it automatically falls back to a verbose, line-by-line text progress bar when a non-TTY environment is detected? Ideally, I would like to see real-time status updates (preferably per chain) in the redirected SLURM .out file as they happen.

Thank you for your time and for the incredible work you do.

Best regards,
Jiho

2

We don’t issue log statements during sampling. Here is a hackish way:

import pymc as pm
from pymc.progress_bar import ProgressBarManager

old_update = ProgressBarManager.update

def new_update(self, chain_idx, is_last, draw, tuning, stats): 
    print(chain_idx, draw)  # Do whatever you want with this info
    old_update(self, chain_idx, is_last, draw, tuning, stats)
    
ProgressBarManager.update = new_update

with pm.Model() as m:
    x = pm.Normal("x")
    pm.sample(tune=5, draws=5, chains=2)
3

#8055 introduced the notion of progress bar backends. I also do a lot of model runs on cloud via orchestration, and I was also thinking that a simple “logger backend” would be nice to have. @Kate_Park even if Ricardo’s hack works for you case, if you could open an issue for this feature it’d be very nice.

1 Like
4

Is the output shown in the image above the only feedback provided during the sampling process?

The issue I’m facing is that this output only appears after the sampling is completely finished. In my SLURM environment, I cannot see the progress bar while the task is actually running. As a result, my MCMC sampling has been running for five days now; although top shows high CPU usage across multiple cores, my log files remain completely empty.

Are there alternative methods or diagnostic tools to verify that the sampling is progressing correctly? I need a way to confirm it is working properly before I decide whether to terminate the process.

I would appreciate your guidance, as I am not a computer science expert and am still learning how to navigate these environments.

Thank you!

5

You have all the information that the progress bar has, including sampler stats. But for your case you can check the rate of draws coming in and assess if that’s reasonable. There’s no real statistic for “stuck in an infinite loop”.

My snippet just showed how you could access those in real time and print/log them as you see fit.