# Using multiple datasets to get a single parameter estimation

**URL:** <https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358>\
**Category:** v5\
**Tags:** modeling\
**Created:** [January 9, 2025, 3:17pm UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358 "2025-01-09T15:17:55Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![annavv](https://avatars.discourse-cdn.com/v4/letter/a/e95f7d/32.png) [@annavv](https://discourse.pymc.io/u/annavv)\
**Post date:** [January 9, 2025, 3:17pm UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/1 "2025-01-09T15:17:55Z")

</div>

Hi,

I am working on the following problem:

- I have 10 datasets (lets say they contain measurements of a X and a Y value and each dataset has a different length).
- I want to use pymc to infer the parameters of a model. Lets assume I do a simple linear regression as in the GLM: linear regression notebook [glm-linear](https://www.pymc.io/projects/docs/en/stable/learn/core_notebooks/GLM_linear.html#glm-linear)) .
- The envisioned result is one estimation of the slope and one estimation of the intercept based on the model and 10 datasets.
- I am using the example notebook ([using-data-containers-to-mutate-data](https://www.pymc.io/projects/examples/en/latest/fundamentals/data_container.html#using-data-containers-to-mutate-data)) as a reference to fit the same model to several datasets.
  - But this gives me a list with 10 “idatas” with 10 estimations and thus 10 slopes and 10 intercepts.

What methodology could use to combine the results to get a best guess of the parameters?

Side mark:  
_I was looking into the part of named dimensions with data containers ([named-dimensions-with-data-containers](https://www.pymc.io/projects/examples/en/latest/fundamentals/data_container.html#named-dimensions-with-data-containers)) to see if I can modify this method to make my problem fit into this format, but my datasets have different sizes and no overlapping X values._

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [January 9, 2025, 7:12pm UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/2 "2025-01-09T19:12:37Z")

</div>

`pm.set_data` is not what you want (as you already found). If you just have tabular data, you should just stack it all up and run a single model. If you don’t, I’d need more information to recommend an approach.

---

<div class="post-metadata">

**Author:** ![annavv](https://avatars.discourse-cdn.com/v4/letter/a/e95f7d/32.png) [@annavv](https://discourse.pymc.io/u/annavv)\
**Post date:** [January 13, 2025, 8:42am UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/3 "2025-01-13T08:42:35Z")

</div>

Thanks for your response.  
This is what I already thought the solution would be.

About “stacking it all up”: is this the way of working?

Assume the model for which i do parameter a fit is nonlinear and looks like this:  
`Y = some_non_linear_model(X, param_a, param_b)`

and the code to setup the pymc model looks something like this:

```auto
model = pm.Model()

with model:
    param_a = pm.Normal("param_a", mu=1, sigma=2)
    param_b = pm.Normal("param_b", mu=1, sigma=2)
    
    # Expected value of outcome
    mu = some_non_linear_model(x_obs, param_a, param_b)

    # Likelihood (sampling distribution) of observations
    Y_obs = pm.Normal("Y_obs", mu=mu, sigma=0.5, observed=y_obs)

```

I can get this to work with 1 single `x_obs` and a list of y’s:` y_obs = [y_1, ..., y_10]`  
But i am not sure how to handle `x_obs` (being a list of x\_1 up to x\_10 with all different lengths) and thus `mu` to work with multiple observations.

---

<div class="post-metadata">

**Author:** ![jessegrabowski](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/jessegrabowski/32/5010_2.png) [@jessegrabowski](https://discourse.pymc.io/u/jessegrabowski)\
**Post date:** [January 13, 2025, 8:50am UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/4 "2025-01-13T08:50:33Z")

</div>

You are going to have to be more concrete about the shapes involved in your problem. If you have two datasets (X\_1, y\_1) and (X\_2, y\_2) with shapes (n\_1 \times k), (n\_1,) and (n\_2 \times k), (n\_2, ), then you simply form `X = np.concat([X_1, X_2], axis=0)`, `y= np.concat([y_1, y_2], axis=0)` and then `mu = f(X, a, b)` has shape (n\_1 + n\_2, ).

---

<div class="post-metadata">

**Author:** ![annavv](https://avatars.discourse-cdn.com/v4/letter/a/e95f7d/32.png) [@annavv](https://discourse.pymc.io/u/annavv)\
**Post date:** [January 13, 2025, 10:08am UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/5 "2025-01-13T10:08:47Z")

</div>

Thanks, I think I was making it too complex… I will try if this works and will come back to that .

I might need some shifting and re-indexing of my timeseries to make it work.

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [January 13, 2025, 10:35am UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/6 "2025-01-13T10:35:22Z")

</div>

You can also just keep the logic separate and add 10 observed variables without trying to concatenate or stack them. If they are using the same unobserved variables the information will flow the same way and you get the posterior corresponding to the multiple evidences

---

<div class="post-metadata">

**Author:** ![annavv](https://avatars.discourse-cdn.com/v4/letter/a/e95f7d/32.png) [@annavv](https://discourse.pymc.io/u/annavv)\
**Post date:** [January 13, 2025, 11:15am UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/7 "2025-01-13T11:15:46Z")

</div>

That is actually the most easiest way of making this work. And with about 10 observations this is also not that much of an effort.

Thanks for your help 👍

This is how I did interpret your last suggestion

```auto

  # Expected value of outcome
  mu1 = some_non_linear_model(X_obs[0], a, b)
  mu2 = some_non_linear_model(X_obs[1], a, b)

  # Likelihood (sampling distribution) of observations
  Y_obs = pm.Normal("Y_obs", mu=mu1, sigma=1, observed=observed_data[0])
  Y_obs2 = pm.Normal("Y_obs2", mu=mu2, sigma=1, observed=observed_data[1])

```

---

<div class="post-metadata">

**Author:** ![ricardoV94](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/ricardov94/32/5775_2.png) [@ricardoV94](https://discourse.pymc.io/u/ricardoV94)\
**Post date:** [January 13, 2025, 11:32am UTC](https://discourse.pymc.io/t/using-multiple-datasets-to-get-a-single-parameter-estimation/16358/8 "2025-01-13T11:32:48Z")

</div>

Yes