AIC or BIC for map estimates

Questions v5
1

I am trying to find examples of how to retrieve different model-comparison metrics using the map estimates in pymc4 using the find_map function.
Mainly i am looking for metrics that penalizes a model with great predictive power if it has a large number of latent variables.
Couldnt find any in the examples section.

kind regards

2

There are no examples for AIC/BIC because they aren’t fully Bayesian evaluation criteria – the assumptions built into using them are quite strong, and if you estimate the full posterior, you have access to nicer metrics (LOO).

If you’re dedicated to just using point estimates though, you can easily compute AIC or BIC just using their formulas. For example, AIC is 2k - 2 \log(L), where k is the number of parameters and L is the likelihood of the data given the model.

For example, here’s an AR(3) model lifted from the statsmodels documentation:

with pm.Model() as model:
    rho = pm.Normal('rho', sigma=100, size=4)
    obs = pm.AR('obs', rho=rho, constant=True, observed=housing)
    map_est = pm.find_MAP()

map_est is a dictionary of variable name:estimated value key-value pairs, so we can use it as input to the compiled log probability function to compute L in the AIC formula:

f_logp = model.compile_logp()
# k is the total number of estimated parameters
k = sum(list(map(len, map_est.values())))
aic  = 2 * k - 2 * f_logp(map_est)

BIC is k \log n - 2 \log L, and can be computed in a similar way.

All this being said, I strongly recommend you do full posterior estimation and use LOO to do model selection if you can.

1 Like
3

Careful, with transformed parameters find_map returns the transformed and untransformed views (and possibly deterministics as well?) so you would double count parameters like this. I suggest using the dictionary returned by model.initial_point() instead.

2 Likes
4

Good to know! I don’t work with MAP much, and only tested my solution on simple examples (no transforms)

2 Likes