Regarding Variational Inference Project for GSoC 2020

Development
1

Hi
I am Sayam Kumar. I have been going through the list of GSoC projects for PyMC this year. I am interested in working on Variational Inference project. Getting familiar with the codebase of PyMC4, I solved a few issues. Can someone guide me how to get started?

@lucianopaz Could you help?

1 Like
2

I’m not sure if I’ll be available to be a mentor through the summer. @ferrine is better suited for this.

3

Hi Sayam,

Good to hear that you are interested and the way you started contributing to PyMC4 is excellent! We can figure out the mentoring separately, I think the main step is in writing a short proposal (which we can review) and formally apply to GSoC.

Best,
Thomas

4

Thanks @lucianopaz and @twiecki for the reply. I will write a proposal for the Variational Inference project and start looking upon its implementation in PyMC3.

5

Hi, Sayam!

So far the PyMC3 VI design is complicated to implement it from the start. It relies on OPVI paper https://arxiv.org/abs/1610.09033 that is a very general theoretical VI framework.

I would suggest to focus the proposal on ADVI and FullRank ADVI and not look in depth of PyMC3. The latter (FR ADVI) would require flattened view of the parameter space and I guess more time to implement and debug. You can send the proposal earlier that the deadline so we can review it and polish!

6

Thanks @ferrine for your suggestions.

7

Just curious to know if there is any active IRC or slack channel for PyMC?

8

Hi
I have created the first draft of my proposal. Github Link. Requesting @lucianopaz and @ferrine for review and suggestions. As the time progesses, I will make the implementation design less and less abstract.
While going through the implementation of OPVI in PyMC3, it heavily uses symbolic PyMC. Any guidelines to understand its use?

9

Hi, Sayam!

I’ve checked your proposal. I still suggest not taking OPVI way for PyMC4 GSoC. There are a couple of reasons for me to be skeptical about it:

  1. Implementation was hard and took around a year of continuous development and achieving speed parity with its previous functional implementation (for a very long time we had legacy pm.advi because of performance reasons). For this reason, I see OPVI way to be a dead way. I hardly believe it would be doable in 3 months.
  2. Given point one is not a concern and PyMC3 OPVI implementation is cristal clear, TF2 lacks core abstractions I relied on during my work on OPVI. Namely, symbolic graph representation, theano.clone, (easy) vectorized input to graph. Without these tools/properties it would be challenging to “just implement” the concept.
  3. Given the above points are still not a concern, what I doubt, there is yet no need to put a lot of abstractions for the inner implementation. You may see, OPVI in PyMC3 is working more like building blocks for VI (KL, Stein, Amortized). We yet have no blocks in PyMC4. That is why there is still no need for that level of abstractions.

As for PyMC4 project, we need to create a simple API, similar to pm.fit to provide few VI implementations, such as MeanField/FullRank ADVI. So far, the top abstraction level is an Approximation. And still, TF2 has its caveats, VI implementation will require a sort of graph manipulation which we only can access in a too limited way. I see this level as minimal required to implement VI, but modifications to pm4 Executor and its logic are very likely

I hope we’ll get the need in OPVI, but not in the nearest future. Anyway, Approximation is a part of OPVI and the future work may start out of this milestone (that is a GSoC project).

I suggest rewriting the proposal narrowing the scope for MeanField and FullRank ADVI only.

Best, Max

2 Likes
10

Thanks @ferrine for sharing your experience with OPVI. I will rewrite the proposal considering MeanField and Full Rank ADVI only.

11

Hi @ferrine
I have modified the initial draft of my proposal to account for MeanField and FullRank ADVI. Github Link. It would be great to hear your suggestions and any guidelines to solve for challenges described in the proposal.

Thanks
Sayam Kumar