# Strange behaviour for out of sample predictions

**URL:** <https://discourse.pymc.io/t/strange-behaviour-for-out-of-sample-predictions/14678>\
**Category:** v5\
**Tags:** modeling\
**Created:** [June 26, 2024, 11:02am UTC](https://discourse.pymc.io/t/strange-behaviour-for-out-of-sample-predictions/14678 "2024-06-26T11:02:04Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![blazysecon](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/blazysecon/32/6152_2.png) [@blazysecon](https://discourse.pymc.io/u/blazysecon)\
**Post date:** [June 26, 2024, 11:02am UTC](https://discourse.pymc.io/t/strange-behaviour-for-out-of-sample-predictions/14678/1 "2024-06-26T11:02:04Z")

</div>

Hi all. I have the following model to pull estimated metric deltas or lifts (percentage impact) for a/b tests (experiments), with very intuitive results, where experiment lifts, which are more uncertain are pulled towards zero, while more certain experiment lifts are kept as is (see included chart)

```auto
experiment_ids = np.array([f"exp{i+1:>02.0f}"for i in range(30)])
experiment_lifts = np.array([-0.02, -0.05, -0.011, -0.011, -0.011, -0.021, -0.013, -0.01, -0.0011, -0.0021, -0.0031, -0.0021, -0.0011, -0.0001, -0.0001, -0.0001, 0.001, 0.002, 0.003, 0.003, 0.003, 0.003, 0.01, 0.01, 0.012, 0.013, 0.013, 0.015, 0.02, 0.05])
standard_errors = np.array([0.002, 0.0002, 0.02, 0.01, 0.003, 0.002, 0.0002, 0.02, 0.01, 0.003, 0.002, 0.0002, 0.02, 0.01, 0.003, 0.002, 0.0002, 0.02, 0.01, 0.003, 0.002, 0.0002, 0.02, 0.01, 0.003, 0.002, 0.0002, 0.02, 0.01, 0.003,])

with pm.Model() as normal_normal_non_centered_model:
    normal_normal_non_centered_model.add_coord("experiment", experiment_ids)

    lifts = pm.Data("lifts", experiment_lifts, dims="experiment")
    standard_errors = pm.Data(
        "standard_errors", standard_errors, dims="experiment"
    )

    M = pm.Normal("M", mu=0, sigma=0.02)
    s = pm.HalfNormal("s", 0.02)
    noise = pm.Normal(
        "noise", mu=0, sigma=1, dims="experiment"
    )  
    delta = pm.Deterministic("delta", M + s * noise, dims="experiment")
    obs = pm.Normal(
        "obs", mu=delta, sigma=standard_errors, observed=lifts, dims="experiment"
    )

with normal_normal_non_centered_model:
    idata = pm.sample(tune=15000, target_accept=0.99)

with normal_normal_non_centered_model:
    idata.extend(pm.sample_posterior_predictive(idata))

posterior_pred_samples = idata.posterior_predictive["obs"]
posterior_pred_mean = posterior_pred_samples.mean(("chain", "draw"))
posterior_pred_se = posterior_pred_samples.std(axis=(0, 1))

```

 ![Screenshot 2024-06-26 at 13.56.51](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/6/65b6373ed72ea89e02f185b7dbddfa08f623cd63.png)

However, when I try out of sample prediction, the posterior predictions are all centered around zero and behave very differently:

```auto
new_experiment_ids = np.array(["new_exp1", "new_exp2", "new_exp3", "new_exp4"])
new_lifts = np.array([-0.02, 0.01, 0.02, 0.05])
new_standard_errors = np.array([0.002, 0.0002, 0.02, 0.003])

# Define the new coordinates
new_coords = {"experiment_variant": new_experiment_ids}

with normal_normal_non_centered_model:
    # Update the dimensions to match new data
    pm.set_data({"lifts": new_lifts, "standard_errors": new_standard_errors}, coords=new_coords)

    # Draw posterior predictive samples using the updated model with new data
    out_of_sample_predictions = pm.sample_posterior_predictive(idata, predictions=True)

posterior_pred_samples = out_of_sample_predictions.predictions["obs"]
posterior_pred_mean = posterior_pred_samples.mean(("chain", "draw"))
posterior_pred_se = posterior_pred_samples.std(axis=(0, 1))

```

 ![Screenshot 2024-06-26 at 13.57.00](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/d/d9d471697cdc4e262c75ba75cfbe7d95a13d34a8.png)

Is it an issue with model specifications, or I am doing out of sample predictions incorrectly? Thanks a loft for you support!

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [June 26, 2024, 12:30pm UTC](https://discourse.pymc.io/t/strange-behaviour-for-out-of-sample-predictions/14678/2 "2024-06-26T12:30:33Z")

</div>

Sounds similar to [BUG: Using a length-only coord causes it to be volatile · Issue #7376 · pymc-devs/pymc · GitHub](https://github.com/pymc-devs/pymc/issues/7376)

---

<div class="post-metadata">

**Author:** ![blazysecon](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/blazysecon/32/6152_2.png) [@blazysecon](https://discourse.pymc.io/u/blazysecon)\
**Post date:** [July 7, 2024, 4:21pm UTC](https://discourse.pymc.io/t/strange-behaviour-for-out-of-sample-predictions/14678/3 "2024-07-07T16:21:39Z")

</div>

> Sounds similar to [BUG: Using a length-only coord causes it to be volatile · Issue #7376 · pymc-devs/pymc · GitHub](https://github.com/pymc-devs/pymc/issues/7376)

@ricardoV94 I can see that this bug has already been fixed, but it did not address my issue. It’s probably something else, like the way I use new (out of sample data) or the model specification issue. Thanks!