I am using a MatrixNormal distribution in my model: obs=pm.MatrixNormal('obs',mu=mean,rowchol=R_chol,colchol=cross_chol,observed=df,shape=df.shape) The dataframe df contains incomplete data, as a lot of its entries are None. As suggested by https://nbviewer.jupyter.org/github/fonnesbeck/scipy2014_…

Your error is actually happening when trying to convert the data to InferenceData. PyMC3 now delegates plotting, stats and diagnostics to ArviZ, it still offers the plotting functions as pm.function but these are aliases to ArviZ functions exposed also from the pymc3 namespace for convenience. You …

Thanks a lot for your response. I was using PyMC3 v3.8 at that time. I just updagred to v3.11.1 and everything works great. By the way, the version upgrade lead to an impressive speed increase. Was this to be expected?

There have been many improvements since v3.8 so it’s not unexpected you now see speed-ups somewhere. What I can’t know though (because I don’t remember all the changes since 3.8 nor I know your model) is where these speed-ups will have effect. Is it during sampling? Plotting?

I actually noticed the speed-up during sampling. In a simple test model, inference used to take ~60 seconds with v3.8, whereas with v3.11.1 it takes ~40 seconds. What’s even more important is that v3.11.1 can handle a large model with lots of incomplete observations, where v3.8 used to crash.

If you have a large number of observations and don’t plan on doing model comparison with some of az.compare, az.loo or az.waic at all you can accelerate the conversion process by using idata_kwargs={"log_likelihood": False} in the pm.sample call if you are not doing so already. The main improvement …

Thanks for your advice, I will start using this statement. Indeed, there was no difference in speed, but I guess that the imporved memory usage will help, especially in larger models.

Just for mentioning, my model still crashes when the number of missing obseravtions becomes too large. The message I get is: The derivative of RV obs_missing.ravel()[7576] is zero., for all my obs_missing variables. With smaller numbers of missing observations, v3.11.1 works well, as mentioned previ…

I don’t really know much about the missing variable imputation internals. Maybe this talk helps better understand what is going on [image] Partial Missing Multivariate Observation and What to Do With Them by Junpeng Lao PyMCon2020 Talk Abstract Missing value i…

FWIW I recently included missing values in a model I’m building. I followed junpeng’s example that Oriol noted above and everything works as expected: I think the most important aspects are standardizing the dataset and using a hierarchical prior on the missing data. I’ve an example here as part of…

Pm.traceplot not working when the observations are incomplete

Questions

OriolAbril April 5, 2021, 6:36pm 9

I don’t really know much about the missing variable imputation internals. Maybe this talk helps better understand what is going on

Topic		Replies	Views
Az.plot_trace() gets an "No module named 'pymc3'" Error v5 bug , arviz	2	645	February 7, 2023
Plotting a trace potentially causes my Python kernel to crash? Questions	9	1834	February 26, 2020
Pm.forestplot and pm.traceplot very slow Questions	6	2093	April 2, 2020
Pm.sample result does not work with az.plot_trace or other az functions v5	9	1566	June 22, 2022
Dealing with missing data and custom distribution Questions	13	2359	March 14, 2021

Pm.traceplot not working when the observations are incomplete

Related topics