Az.loo returns nan

Questions version agnostic
,
1

Hi everyone,

I estimate a hierarchical model on 200 decisions from 300 subjects with 4 types of parameters (with a subject and a condition level) and everything looks great so far: I get R-hat of 1 (sometimes 1.01) on all parameters and only 1 divergence popped up during estimation.

However, I have the following problem: The LOO calculation fails. I get many RuntimeWarnings such as "

  • divide by zero encountered in divide b_ary [/](https://file+.vscode-resource.vscode-cdn.net/)= prior_bs * ary[int(n [/](https://file+.vscode-resource.vscode-cdn.net/) 4 + 0.5) - 1]
    or
  • divide by zero encountered in divide b_ary += 1 [/](https://file+.vscode-resource.vscode-cdn.net/) ary[-1]
    and ultimately the following UserWarning:
  • UserWarning: Estimated shape parameter of Pareto distribution is greater than 0.7 for one or more samples. You should consider using a more robust model, this is because importance sampling is less likely to work well if the marginal posterior and LOO posterior are very different. This is more likely to happen with a non-robust model and highly influential observations.

Conversely, The Output results in “nan”:

Computed from 12000 posterior samples and 60000 observations log-likelihood matrix.

         Estimate       SE
elpd_loo      nan      nan
p_loo         nan        -

There has been a warning during the calculation. Please check the results.
------

Pareto k diagnostic values:
                         Count   Pct.
(-Inf, 0.5]   (good)     54086   90.1%
 (0.5, 0.7]   (ok)        1762    2.9%
   (0.7, 1]   (bad)       1638    2.7%
   (1, Inf)   (very bad)  2514    4.2%

How do I proceed from here? The WAIC calculation works, but gives me a similar Warning:

UserWarning: For one or more samples the posterior variance of the log predictive densities exceeds 0.4. This could be indication of WAIC starting to fail. 
See http://arxiv.org/abs/1507.04544 for details
  warnings.warn(
Computed from 12000 posterior samples and 60000 observations log-likelihood matrix.

          Estimate       SE
elpd_waic -14359.38   133.33
p_waic      629.21        -

There has been a warning during the calculation. Please check the results.

What do you recommend? Thank you!

Best
N

2

Seeing as you get nan for the summary but it is still possible to bin the Pareto k diagnostics, my guess is there is one (or a few) problematic observations. I would inspect the loo.loo_i values to see how many and which are nans. Best case scenario, we can narrow down the issue with this info, worst case scenario, we compute the loo estimate excluding those observations (also for any models we want to compare with) which is not ideal but should give the same results if there is really nothing particular about those observations that would explain the appearance of nans.