# How to add weights to data in bayesian linear regression

**URL:** <https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362>\
**Category:** Questions\
**Tags:** linear\_model\
**Created:** [November 26, 2021, 7:37am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362 "2021-11-26T07:37:52Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanket](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sanket](https://discourse.pymc.io/u/sanket)\
**Post date:** [November 26, 2021, 7:37am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/1 "2021-11-26T07:37:52Z")

</div>

I have implemented a bayesian model like this:

```auto
 #Bayesian Linear Regression
    with pm.Model() as predictive_model:
        x = pm.Data("x",xtrain.values)
        y = pm.Data("y",ytrain.values)
        a = pm.Normal('slope', mu=0, sd = 16)
        b = pm.Normal('intercept',mu= 0, sd = 16)
        s = pm.HalfNormal('error',sd =1)
        y_pred = a*x+b

        obs = pm.Normal('observation', mu = y_pred, sd=s, observed=y)

        start = pm.find_MAP()
        step = pm.NUTS(scaling=start)
        trace = pm.sample(1000, step, start=start, progressbar=True)

```

Now, I need to add weights associated with each data point, weights are based on recency of data(recent data is given more weights and this weight is already available to me as an array). how can I assign them?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [November 26, 2021, 3:54pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/2 "2021-11-26T15:54:21Z")

</div>

Welcome!

There are various ways to incorporate weights into your model. Many will write custom logp functions (e.g., [here](https://discourse.pymc.io/t/how-to-run-logistic-regression-with-weighted-samples/5689) and [here](https://discourse.pymc.io/t/how-to-run-logistic-regression-with-weighted-samples/5689/5)). But I personally prefer to adjust the scale parameter of my observed variable, which allows for more heavily weighted observations to contribute more model’s logp. It’s simple and relatively transparent. Here’s a simple example:

```python
s = pm.HalfNormal('error', sd = 1)
scaled_s = pm.Deterministic('scaled_s', s / y_weights)
obs = pm.Normal('observation',
                mu = y_pred,
                sd=scaled_s,
                observed=y)

```

In addition, I would strongly suggest using the defaults when calling `pm.sample()` until you are confident that they don’t work for you. I would particularly suggest avoiding the use of `pm.find_MAP()`

```python
# nice and simple
trace = pm.sample(1000)

```

---

<div class="post-metadata">

**Author:** ![sanket](https://avatars.discourse-cdn.com/v4/letter/s/57b2e6/32.png) [@sanket](https://discourse.pymc.io/u/sanket)\
**Post date:** [November 29, 2021, 6:01am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/3 "2021-11-29T06:01:11Z")

</div>

@cluhmann Thanks. I added the weights but I am getting an error while running the sample\_posterior\_predicive on test set

```auto
with predictive_model:
     32 pm.set_data(dict(x=xtest.values,y=ytest.values))
---> 33 posterior = pm.sample_posterior_predictive(trace)
.
.
ValueError: Cannot broadcast provided shapes (7,), (73,), () given size: ()

```

This issue doesn’t occur when I don’t use weights.  
my train data set has 73 data points with 1 feature and test set has 7 data points.

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [November 29, 2021, 2:59pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/4 "2021-11-29T14:59:04Z")

</div>

Yes, adding the weights integrates the data more deeply into the model and makes it harder to just swap out data sets. In these cases, I tend to use the samples in the trace to generate whatever I need “by hand”. Something along these lines:

```python
import matplotlib.pyplot as plt
n_posterior_samples = 100
for _ in range(n_posterior_samples):
    sample = np.random.choice(trace)
    mean_hat = sample['intercept'] + (sample['slope'] * new_x)
    obs_hat = np.random.normal(loc=mean_hat,
                               scale=sample['error'] / new_weights)
    plt.plot(new_x, obs_hat)
plt.show()

```

---

<div class="post-metadata">

**Author:** ![Chandan\_Gupta](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/chandan_gupta/32/4736_2.png) [@Chandan\_Gupta](https://discourse.pymc.io/u/Chandan_Gupta)\
**Post date:** [January 23, 2022, 11:21am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/5 "2022-01-23T11:21:29Z")

</div>

Hi, I was actually going through a similar problem, and in my case the accepted solution will be very inefficient since I need to also update the model in an online manner. Instead, I found a better way to do the same by specifying the weights in a data container, and during inference with new data, simply set the data container value with test dataset :

```auto
s = pm.HalfNormal('error', sd = 1)
weights = pm.Data("weights", y_weights) 
scaled_s = s/weights
obs = pm.Normal('observation',
                mu = y_pred,
                sd=scaled_s,
                observed=y)

```

After training through this, during inference, when you set your test data, simple update the data container by

```auto
pm.set_data("weights", y_weights_test)

```

You should be able to run it now.  
The problem with pm.Deterministic() call is that it makes each weight a parameter and therefore its size become fixed.

Let me know if you have any further questions, or any feedback for this trick.

---

<div class="post-metadata">

**Author:** ![mcao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/mcao/32/4886_2.png) [@mcao](https://discourse.pymc.io/u/mcao)\
**Post date:** [August 17, 2022, 2:27am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/6 "2022-08-17T02:27:28Z")

</div>

Question with how much impact scaling a parameter has on the model.

If `s` is the learnable parameter and we set `sd=scaled_s`, then wouldn’t `s` be skewed towards a smaller value anyway? If yes, then wouldn’t that reduce the impact scaling has?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [August 17, 2022, 3:46am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/7 "2022-08-17T03:46:29Z")

</div>

> [@mcao](#):
>
> wouldn’t `s` be skewed towards a smaller value anyway?

It’s not clear what you mean by this. Skewed in what way?

---

<div class="post-metadata">

**Author:** ![mcao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/mcao/32/4886_2.png) [@mcao](https://discourse.pymc.io/u/mcao)\
**Post date:** [August 18, 2022, 2:11am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/8 "2022-08-18T02:11:29Z")

</div>

Sorry if I was vague about this. Let me give an example.

Let’s say you have 100 response data \boldsymbol{y} that are from two different source. One from simulation y\_{sim}, and another using real life experiments y\_{exp}. The size of the response data from each source is \text{dim}(y\_{sim}) = 95, and \text{dim}(y\_{exp}) = 5. And you also want to lower weight (reduce the importance) of the simulation data, and increase the weight for the real experiments.

The standard deviation on the simulation data is very small, s\_{sim} = 0.01, compared to the real experiments, s\_{exp} = 0.30.

If we have two models. The first model doesn’t weight the data, and the second model does. And let’s say the weights for simulation data is 0.05, and real experiment data is 1. Below are code chunks below for the two models.

```auto
# model 1
s1 = pm.HalfNormal('error', sd = 1)
obs = pm.Normal('observation',
                mu = y_pred,
                sd=s1,
                observed=y)

```

```auto
# model 2
s2 = pm.HalfNormal('error', sd = 1)
weights = pm.Data("weights", y_weights) # w_sim = 0.05, w_exp = 1
scaled_s = s2/weights
obs = pm.Normal('observation',
                mu = y_pred,
                sd=scaled_s,
                observed=y)

```

Wouldn’t the `s2` posterior be much much smaller than `s1` because majority of the data in \boldsymbol{y} has a small standard deviation? If `s2` posterior will be much smaller than `s1`, then `scaled_s` will be similar to `s1`. Therefore, scaling the data isn’t that effective.

Any thoughts on this?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [August 18, 2022, 3:19pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/9 "2022-08-18T15:19:29Z")

</div>

> [@mcao](#):
>
> Wouldn’t the `s2` posterior be much much smaller than `s1` because majority of the data in \boldsymbol{y} y\boldsymbol{y} has a small standard deviation? If `s2` posterior will be much smaller than `s1`, then `scaled_s` will be similar to `s1`. Therefore, scaling the data isn’t that effective.

Yes, that all seems correct to me. The value of `scaled_s` is likely to be similar to `s1` _on average_. But the weighting scheme isn’t intended to do anything on average, it’s intended to weigh some observations more than others. So it’s effect is relative, not absolute. Maybe it helps to reformulate your model 1 like this:

```python
# model 1
s1 = pm.HalfNormal('error', sd = 1)
weights = pm.Data("weights", np.ones(size=N))
scaled_s = s1/weights
obs = pm.Normal('observation',
                mu = y_pred,
                sd=scaled_s,
                observed=y)

```

This is identical to your model 2, but with equal weights applied to all observations which is the weighting implied by your original model 1. The estimates of `s1` should be identical to your original model 1.

Now we can do the following:

```python
# model 1
s1 = pm.HalfNormal('error', sd = 1)
weights = pm.Data("weights", 100*np.ones(size=N))
scaled_s = s1/weights
obs = pm.Normal('observation',
                mu = y_pred,
                sd=scaled_s,
                observed=y)

```

Here, we have the same, uniform weighting scheme, but we associate every observation with a weight of **100** , instead of the previous weight of **1**. The posterior of `s1` will likely suggest larger values in order to compensate for the extremely large weights we have applied. But things should mostly balance out.

But this is very different than the following, which is basically the idea suggested in the original answer:

```python
# model 1
s1 = pm.HalfNormal('error', sd = 1)
weights = pm.Data("weights", [100,100,1,1,100,1,100,1,1,1,100])
scaled_s = s1/weights
obs = pm.Normal('observation',
                mu = y_pred,
                sd=scaled_s,
                observed=y)

```

Here, the weights vary across observations. The sampled values of `s1` will try to balance the variance in the observed data (`y`), but must accommodate the fact that some weights are large and some are small.

Does that help?

---

<div class="post-metadata">

**Author:** ![mcao](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/mcao/32/4886_2.png) [@mcao](https://discourse.pymc.io/u/mcao)\
**Post date:** [August 19, 2022, 3:38am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/10 "2022-08-19T03:38:06Z")

</div>

Thanks @cluhmann. If I understood your example above correctly, we would use weight of 100 for  
the more interesting/important observed data like y\_{exp} and the smaller weight of 1 for y\_{sim}? The larger weight decreases `scaled_s` for the interesting/important observed data like y\_{exp}, allowing these observed data to have more influence on the posterior. And the smaller weight increases `scaled_s` for y\_{sim}. The `s1` parameter may be somewhere between very large and small value to balance the overall variance of \boldsymbol{y}. Is my understanding here correct?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [August 19, 2022, 3:55am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/11 "2022-08-19T03:55:26Z")

</div>

Exactly!

---

<div class="post-metadata">

**Author:** ![mrkw](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mrkw](https://discourse.pymc.io/u/mrkw)\
**Post date:** [January 30, 2023, 8:07pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/12 "2023-01-30T20:07:39Z")

</div>

Thanks for this super helpful thread @mcao and @cluhmann! I used this weighting “trick” in a hierarchical model and I noticed after adding in the weighting to the sigma, the varying intercept effects significantly changed in scale.

Is there a change in interpretation when adding weights for varying intercepts?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [January 30, 2023, 9:24pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/13 "2023-01-30T21:24:53Z")

</div>

> [@mrkw](#):
>
> adding weights for varying intercepts?

Can you say more about what you mean by this? Do you simply have a model with varying intercepts and that you have used the weighting scheme described above with this model? Or do you mean that you added “for varying intercepts” in some more specific way?

---

<div class="post-metadata">

**Author:** ![mrkw](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mrkw](https://discourse.pymc.io/u/mrkw)\
**Post date:** [January 30, 2023, 9:54pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/14 "2023-01-30T21:54:32Z")

</div>

Sorry for the unclear description! The former! The model has varying intercepts (or hierarchical effects or whatever nomenclature that you’d use). Here’s a simplified, high level view of the model:

```py
with pm.Model(coords=COORDS) as model:
    #global intercept
    intercept = pm.Normal('intercept', mu = 0, sigma = .15)
    
    # hyper-prior varying intercept
    indv_mean = pm.Normal("indv_mean", mu=0, sigma= .05)
    indv_sigma = pm.HalfNormal("indv_sigma", sigma = .15)

    #indv idx
    indv_idx = pm.ConstantData("indv_idx", indv_codes, dims="obs")
    
    #non-centered param
    indv_offset = pm.Normal('indv_offset', mu=0, sigma=1, dims = "obs") 
    indv= pm.Deterministic("imdv", indv_mean + (indv_offset * indv_sigma ))

    indv_eff = indv[indv_idx]

    #Covs
    covariates = pm.ConstantData('covariates', df.covariates, dims = "obs")
    b = pm.Normal("b", mu=0, sigma=.05)
    
    weights = pm.ConstantData('weights', df.weight.values, dims = "obs")

    # Model error:
    sigma = pm.HalfNormal("sigma", 20)
    
    scaled_sigma = pm.Deterministic('scaled_sigma', sigma / weights)
    
    mu = pm.Deterministic("mu", 
    intercept + (b * covariates) + indv_effect
    )
    y = pm.Normal("y", mu, sigma=scaled_sigma, observed= df.observed )

```

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [February 1, 2023, 2:30pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/15 "2023-02-01T14:30:45Z")

</div>

I don’t see any reason why the scaling “trick” would work in a fundementally different way once you incorporate varying intercepts. Of course, changing one part of the model will always affect the estimates of other parameters, but knowing exactly what consequences to expect is very much dependent on the specifics of your model and your data. I might suggest taking a subset of your data (e.g., a subset small enough to visualize individual data points) and running “weighted” and unweighted models so that you can better investigate how the intercepts and the weights might be related.

---

<div class="post-metadata">

**Author:** ![mrkw](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mrkw](https://discourse.pymc.io/u/mrkw)\
**Post date:** [February 7, 2023, 3:59pm UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/16 "2023-02-07T15:59:02Z")

</div>

Yep, after digging around I think it all makes sense. Thanks @cluhmann!

---

<div class="post-metadata">

**Author:** ![GabrielBMiranda](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/gabrielbmiranda/32/4671_2.png) [@GabrielBMiranda](https://discourse.pymc.io/u/GabrielBMiranda)\
**Post date:** [May 22, 2023, 1:24am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/17 "2023-05-22T01:24:17Z")

</div>

@cluhmann I love the trick you presented here and always adopt this approach for my problems. Do you have any references for this strategy of weighting?

---

<div class="post-metadata">

**Author:** ![cluhmann](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/cluhmann/32/3083_2.png) [@cluhmann](https://discourse.pymc.io/u/cluhmann)\
**Post date:** [May 22, 2023, 4:48am UTC](https://discourse.pymc.io/t/how-to-add-weights-to-data-in-bayesian-linear-regression/8362/18 "2023-05-22T04:48:54Z")

</div>

Not entirely sure where I picked it up. It’s basically a Bayesian version of weighted regression. You can see some discussion on the [weighted least squares](https://en.wikipedia.org/wiki/Weighted_least_squares) Wikipedia page or Gelman & Hill’s _Data Analysis Using Regression and Multilevel/Hierarchical Models_ (e.g., chapter 18). But I’m sure it’s covered elsewhere!