Regularizing the posterior of a parameter with the KL divergence from a different posterior

[bob-carpenter]

@jovank

I have been struggling to implement a simple idea I have for penalizing the likelihood of a model if the posterior of a parameter calculated on a small sample differs from the posterior of that same parameter calculated on a very large sample.

@jovank

Do you think there is a better way to do this?

What you want is called a hierarchical model—I’d call it the bread-and-butter of Bayesian modeling. It doesn’t need KL divergence. Here’s a simple PyMC example:

Hierarchical Partial Pooling

and here’s my longer Stan example along the same lines and using the same classic Efron and Morris data:

Hierarchical Partial Pooling for Repeated Binary Trials

@jovank

Is there a way to somehow use the posterior logp here as well?

The usual way to control the posterior is through the prior, unless you want to change the likelihood.