Hi @Jovan, glad you liked the blog post.
Not quite sure if this will answer your question, but we can imagine two ends of a spectrum.
- On one end we have many A/B tests each with modest numbers of observations. This was pretty much the situation in the blog post you mentioned.
- On the other end we have one (or a few) A/B tests, each with many observations. We also addressed this with a different (unnamed) client written up here Bayesian inference at scale: Running A/B tests with millions of observations - PyMC Labs
So depending where we are on this spectrum, different approaches produce different speed-ups.
Not sure if that helps? Feel free to DM me if you think an engagement with PyMC Labs might help for your particular problem 