# Calculating WAIC/LOO on different size datasets

**URL:** https://discourse.pymc.io/t/calculating-waic-loo-on-different-size-datasets/12764
**Category:** version agnostic
**Tags:** linear\_model, arviz, modeling
**Created:** [August 20, 2023, 4:39pm UTC](https://discourse.pymc.io/t/calculating-waic-loo-on-different-size-datasets/12764 "2023-08-20T16:39:56Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![cocodimama](https://avatars.discourse-cdn.com/v4/letter/c/6a8cbe/32.png) [@cocodimama](https://discourse.pymc.io/u/cocodimama)
#### Post date: [August 20, 2023, 4:39pm UTC](https://discourse.pymc.io/t/calculating-waic-loo-on-different-size-datasets/12764/1 "2023-08-20T16:39:56Z")

</div>

Hi there,

Suppose I have the following two datasets for a simple hierarchical linear regression with no intercept:

- Dataset 1: X1, Y1
- Dataset 2: X2, Y2

X1, X2, Y1 and Y2 are all scalars.

Dataset 2 is a transformed version of Dataset 1, where the X and Y values can be:

1. Different
2. It can also have a different number of observations (due to filtering or other things)

I am reasonably well-versed in using WAIC/LOO for model comparison on a fixed dataset, but would it ever make sense to use WAIC/LOO for model comparison when the datasets are different?

What I’m trying to understand is for which dataset a given specification of linear model is most likely to generalize - for my purposes, I can actually use either and recover the quantity of interest at the end.

For issue 2. I am aware that WAIC/LOO scale with dataset size, but would it be possible to divide by the sample size here to get a kind of normalized estimate? I think this is being done in section 9.3.1 of this book:

[https://bookdown.org/marklhc/notes\_bookdown/model-comparison-and-regularization.html](https://bookdown.org/marklhc/notes_bookdown/model-comparison-and-regularization.html)

For issue 1. I am not sure.

Any help would be much appreciated!

---

<div class="post-metadata">

### Author: ![OriolAbril](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/oriolabril/32/2497_2.png) [@OriolAbril](https://discourse.pymc.io/u/OriolAbril)
#### Post date: [September 27, 2023, 7:49am UTC](https://discourse.pymc.io/t/calculating-waic-loo-on-different-size-datasets/12764/2 "2023-09-27T07:49:54Z")

</div>

I am quite sure the answer is no to both. See for example [Cross-validation FAQ](https://avehtari.github.io/modelselection/CV-FAQ.html#13_Can_cross-validation_be_used_to_compare_different_observation_models __response_distributions__ likelihoods). You can compare transformed versions of the same dataset including the jacobian as shown in [https://oriolabrilpla.cat/en/blog/posts/2019/loo-cv-transformed-data.html](https://oriolabrilpla.cat/en/blog/posts/2019/loo-cv-transformed-data.html) (also another example linked in the faq above) but not if there are different filtering applied to each model.