Hi everyone,
This summer I will be doing some work on the SMC(-ABC) samplers in PyMC3 as part of the GSOC project, with the mentoring of @aloctavodia and @junpenglao. I would love to hear your experiences (if any) with these. What worked for you and what did not, what did you wish was different.
If you never heard of the SMC (Sequential Monte Carlo) and or ABC (Approximate Bayesian Computation), feel free to use this thread to ask any questions about it!
If you wish, you may also check the proposal I wrote. Any feedback is appreciated 
5 Likes
I’m a big fan of ABC and I have used simple rejection, SMC, and more sophisticated approaches. Despite my love of pymc3. I have chosen to implement those projects using other packages. I looked into pymc3’s ABC capabilities (after @aloctavodia mentioned new features were coming), but the primary reason I went elsewhere is that there’s essentially no evidence that pymc3 provides any ABC functionality. The SMC documentation mentions a kernel argument (which defaults to 'metropolis'), but that argument is not described anywhere in the docstring. SMC/ABC is covered in this example notebook, thought these notebooks are always a bit out of date. In that notebook, there is mention of the pm.Simulator() “distribution”, but I can’t find any mention of this anywhere in the API documentation. As a result, it’s not clear how to craft critical ABC ingredients (e.g., summary statistics, distance metrics, thresholds, etc.). To me, the lack of documentation is pretty much fatal.
As a result, I don’t quite know what all pymc3 can do ABC-wise. But I think modularity and flexibility are my general desiderata. Summary statistics and distance metrics obviously, but control over the sampling and adaptation details as well (e.g., I see there is mention of control over mutation rate). I very much like the idea of multiple pm.Simulator() components per model. That would seem to open the door to a vast array of models (e.g., hierarchical models) with little extra effort on the part of the modeler.
I’m looking forward to your work and the code-sharing “contests" you describe. Not sure I can participate without some additional API information, but I would certainly be interested in seeing what others produce.
5 Likes
@cluhmann Thank you so much for your detailed feedback!
Definitely. One of the main goals is indeed to improve the documentation. The pm.Simulator is just a silly case of not being in the .rst files as there are already quite detailed docstrings in the methods. Obviously that is not enough in itself. Anyhow, in addition to improving the standard documentation I am really looking forward to adding some new illustrative (and fun) examples to the Notebook gallery 
I think we share the same desiderata. Both to give a bit more control over the default offers, as well as make it relatively simpler to customize or implement your own ABC algorithm if you wish to. There is no reason why ABC algorithms cannot be used for PyMC3 models, other than lacking a good-enough API for getting things started. Once that’s improved, we will hopefully be in a place to implement “everything” (that is useful) in other libraries while benefiting from the cool things that are unique to PyMC3 and its ecosystem (model building, diagnostics, community, etc…).
Actually those won’t be SMC-ABC specific, but more details… soon
3 Likes