Ive been using PyMC3 with great satisfaction to build my first Bayesian models.
Right now Im working on a simple logistic regression model, which works like a charm.
My next step would be to look at regularizing priors (horseshoe, spike/slab etc) and of course I want to compare the outcomes. In “regular” machine learning I would probably using 10-fold cross-validation, but if I understand correctly PyMC3 can utilize PSIS-LOO-CV to as an alternative, which would be nice since then I only need to fit 1 model instead of 10.
However, I have a decent class imbalance in my set (0.3 vs 0.7), so I am more interested in using the AROC instead of just accuracy to compare.
Is this possible with PyMC3/arviz? What I tried was calling the az.loo function with pointwise=True and then taking the exp of the loo_i array, but this gives me an huge overestimation (AUC 0.89, I wish! Cross validation tells me it should be around 0.7).
Am I doing something wrong? Or is it not possible what I want to do?