AIC or BIC for map estimates

Questions v5
2

There are no examples for AIC/BIC because they aren’t fully Bayesian evaluation criteria – the assumptions built into using them are quite strong, and if you estimate the full posterior, you have access to nicer metrics (LOO).

If you’re dedicated to just using point estimates though, you can easily compute AIC or BIC just using their formulas. For example, AIC is 2k - 2 \log(L), where k is the number of parameters and L is the likelihood of the data given the model.

For example, here’s an AR(3) model lifted from the statsmodels documentation:

with pm.Model() as model:
    rho = pm.Normal('rho', sigma=100, size=4)
    obs = pm.AR('obs', rho=rho, constant=True, observed=housing)
    map_est = pm.find_MAP()

map_est is a dictionary of variable name:estimated value key-value pairs, so we can use it as input to the compiled log probability function to compute L in the AIC formula:

f_logp = model.compile_logp()
# k is the total number of estimated parameters
k = sum(list(map(len, map_est.values())))
aic  = 2 * k - 2 * f_logp(map_est)

BIC is k \log n - 2 \log L, and can be computed in a similar way.

All this being said, I strongly recommend you do full posterior estimation and use LOO to do model selection if you can.

1 Like