AUC with PSIS-LOO-CV

Questions
1

Hello everyone,

Ive been using PyMC3 with great satisfaction to build my first Bayesian models.
Right now Im working on a simple logistic regression model, which works like a charm.

My next step would be to look at regularizing priors (horseshoe, spike/slab etc) and of course I want to compare the outcomes. In “regular” machine learning I would probably using 10-fold cross-validation, but if I understand correctly PyMC3 can utilize PSIS-LOO-CV to as an alternative, which would be nice since then I only need to fit 1 model instead of 10.

However, I have a decent class imbalance in my set (0.3 vs 0.7), so I am more interested in using the AROC instead of just accuracy to compare.

Is this possible with PyMC3/arviz? What I tried was calling the az.loo function with pointwise=True and then taking the exp of the loo_i array, but this gives me an huge overestimation (AUC 0.89, I wish! Cross validation tells me it should be around 0.7).

Am I doing something wrong? Or is it not possible what I want to do?

Many thanks,

2

No one who can help me with this?

3

Hi,

I am not sure I understand the question nor the goal you had in mind. Please correct me if I’m wrong as I’ll assume answers to some of these questions and these assumption may be completely wrong.

I have the feeling you mean root mean square error (or 1-rmse or some related quantity) when you mean accuracy, and that you expect az.loo to use this quantity. However, this is not the case. I’d recommend reading section 2 of “Understanding predictive information criteria for Bayesian models” by Gelman et al.

Regarding RMSE and AROC, az.loo does not work with them, and in addition, I think that PSIS approximation cannot be used in order to estimate the result of cross-validation.