I’m just starting to explore some model comparison with WAIC and LOO (see this notebook). I’m getting an odd error (in cell 14) so I don’t know if this is a bug in pm.compareplot?
PS: I finally got around to starting to port of my Matlab based toolbox to Python. This analyses experimental data from temporal discounting experiments, and is described in this paper. So sorry in advance if my rate of questions asked here increases somewhat!
I think this is already fixed - could you please update to the newest release and try again?
Sure. Still a bit new to python so I’m a bit feeble using conda…
Running conda update --all leaves me with version 3.2. Same with conda update pymc3. I’m not totally clear how to get version 3.3.
conda install -c conda-forge pymc3 seems to have got me version 3.3 
And yes, the issue with LOO is now fixed, other than the warning flags. I wonder how serious they are, or to what extent they can be ignored?
hmm, some of the model is quite simple and should not give a warning. Could you please try just testing the loo for that model as see what warning it gives?
For some of the models (coin flip, exponential, hyperbolic) running pm.loo(trace, model) does not give any warnings at all.
For others (e.g. HyperboloidA) I get
UserWarning: Estimated shape parameter of Pareto distribution is
greater than 0.7 for one or more samples.
You should consider using a more robust model, this is because
importance sampling is less likely to work well if the marginal
posterior and LOO posterior are very different. This is more likely to
happen with a non-robust model and highly influential observations.
happen with a non-robust model and highly influential observations.""")
That does seem to coincide with the The estimated number of effective samples is smaller than 200 for some parameters. warning from the sampling step.
I re-ran with 10,000 samples and still get both errors. Seems a bit odd that effective sample size is <200 given 10,000 samples. Chains are visually fine, Rhat is fine, posterior predictive check confirms a sensible fit. Not quite sure how to progress.
Here’s the model, sitting inside a small class:
Thanks for the help on this by the way
Thanks for reporting back. I think the LOO is catching all error message after we recently change the logging system. I have a PR to fix this but it still needed more work.
As for the effect_n error, they are caused by the Deterministic RVs in your model. For example, if you do the trace plot of Exponential model VA is a flat line. I think we should improve the warning system to catch warning only for the Free RVs. For now you can try doing VA = data['A'] * self._df(k, data['DA']).
I see. So for this particular dataset VA will always be 0. This will not always be the case for other datasets however. Implementing your suggestion to VA would remove the (inappropriate) warning, but it would also restrict the range of datasets that the model would work for.
I think at this point I’ll leave the warning there and just remember that it’s referring to VA, and the warning may magically disappear if you just report the free RV’s (see your new #2822)
I was perhaps thinking that VA and VB don’t need to be explicit nodes in the graph, so you should just inline the self.df() call into the P_chooseB line. But then I remembered, there are situations where you do actually want to sample from VA and VB.
So all in all, I think we’ve found a slight issue - but for my particular models, I’ll leave them as they are and see if the sampling and LOO warnings disappear if #2822 is resolved 
Yep, there would be some inconveniences of not explicitly having them in the trace, but since they are deterministic nodes, you can also compute them after sampling.
Should be fixed now - you can try upgrading to master branch 
Argh, my beginner-with-python status is kicking in. I updated to latest MacOS on this machine and now I can’t run anything. Getting a theano compilation error
You can find the C code in this temporary file: /var/folders/kg/zw9sf78j0bqdjdm4dg4g9lr09zzj75/T/theano_compilation_error_bvz5zzvm
Tried some conda update theano but everything is already up to date
Have to say I have a very beginner understanding of packages and conda etc so I’m a bit stuck. Is there anything I can do which is better than uninstalling/reinstalling anaconda?
Think I need to go watch some tutorial videos on environments and how conda works etc
did you try to uninstall theano and reinstall it again?
So my Matlab background is not serving me well. I’ve had no experience of package/environment management.
I did some conda uninstall theano which also removed pymc3. So then just did conda install -c conda-forge pymc3 but no luck. Tried conda update theano and I just get the same theano compilation error.
Sorry about this, I’ll try to find time to learn more
No problem. My python env set up is also a mess with pip+homebrew and no virtual env…
So i take it there is no error in theano installation? Also did you remove the .theano folder under your home directory in mac?
I didn’t know about the .theano. Gave that a go with no luck. I’ll just do a fresh install because this could go on forever. Thanks for the pointers