Combining multiple likelihoods for astronomical datasets

Questions
1

Branching from existing topic - Calculating WAIC for models with multiple likelihood functions

I would like to consider the possibility of combining multiple log likelihoods for model comparison. Here is the example case that was mentioned by @OriolAbril in the previous thread - say we have TESS and Spitzer points or observations (say 30 and 13 respectively). Since the two instruments can have different systematics, jitter, etc., we would prefer to define separate log-likelihood functions for the different instruments. We would be combining them because the two instruments can have different sensitivities, characteristics, etc., which enable us to probe different regions of the astrophysical parameter space. For example their cadence, or on-sky sensitivity for data gathering might be different. In many cases where we have ground based observations, there can be more than 2 log-likelihood functions as well.

A few potential use cases -

  1. Compare the log-likelihood and fit across different instruments, i.e. does the TESS data fit the astrophysical signal better than the Spitzer data? For a real case, this almost certainly would be the case with different instrumental wavelength coverage.

  2. Combine the instrumental datasets to do model comparison across different runs… This relates to the previous thread where we can use log-likelihood value, WAIC, LOO, bayes factor, bayesian evidence (along with their uncertainties) to infer if one model is significantly preferred to another. It could be in terms of number of planets, circular vs eccentric orbits, or to compare other orbital parameters. This can naively be performed in terms of significance of the posteriors for additional components, however it would also be useful to approach this from a model comparison point of view.

The complication here is that the almost always the different datasets will have different sizes, and in terms of time series will rarely have any overlap, i.e. they will not be contemporaneous observations.

2

Sorry to resurrect an old topic, but this is the top search engine hit when I was trying to solve my specific astronomical dataset problem! I added my solution to the other topic, but I’ll copy/paste it here.

Here, opt.models.values() are my different models, my observations are named 12CN-1 and 12CN-2 with associated coordinates 12CN-1_dim_0 and 12CN-2_dim_0. I rename the observations to spec, rename the coordinate axes to dim_0, then concatenate the two coordinates. The result is a new, single “observation” named 12CN-1, 12CN-2 with coordinate dim_0.

import xarray as xr

for model in opt.models.values():
    model.trace.log_likelihood["12CN-1, 12CN-2"] = xr.concat(
            [
                model.trace.log_likelihood.rename({"12CN-1": "spec"})["spec"].rename({"12CN-1_dim_0": "dim_0"}),
                model.trace.log_likelihood.rename({"12CN-2": "spec"})["spec"].rename({"12CN-2_dim_0": "dim_0"}),
            ],
            dim="dim_0",
        )

Then I can compare models with WAIC/LOO using

az.compare({cloud: model.trace for cloud, model in opt.models.items()}, var_name=“12CN-1, 12CN-2”)

This answers the question: which model has better predictive accuracy for any one datum regardless of the source of that datum.

3

Hi @tvwenger, thank you for reopening this. I didn’t have a satisfactory solution for combining multiple likelihoods and then using that for model comparison (say WAIC or LOO)… perhaps something similar to nested model comparison, where you vary the complexity of different models and use Bayes Factors to evaluate preference.

Which package versions are you using here?

I was trying to achieve something like this, where I could programmatically add likelihoods from X different instruments and then perform a similar model comparison as you suggested.