Design of Experiment - Quantifying Parameter Effects on Response?

Questions
1

Edit: See below for my attempt to build what I described below into a PyMC3 model. I also have a few conceptual questions listed at the bottom of the next post.

Hi everyone! I’m familiar with simple PyMC3 models (ex: coin flips, means comparisons, etc.) but I’m beginning to venture into the more powerful forms of interference.

Specifically, my team ran a design-of-experiment (DoE) matrix where we varied three parameters (A, B, C) and measured the response (T) in an effort to determine the effect of each parameter relative to our current baseline parameters. We ran quite a few baseline experiments in addition to the DoE matrix to build up data.

I’m trying to conceptually think through how I should set up the model in order to tease out these effects. My initial thought was to model the baseline results’ T distribution as a starting point, as the intention is to ultimately measure the effect of changing A, B, and C on this response. Modeling the baseline response alone has an added bonus of measuring the baseline-to-baseline variability (or really, variability inherent to what we’re testing) given fixed parameters.

To summarize up to here - our baseline experiments can be used to understand the variability (ex: standard deviation) between samples tested with our baseline parameters. The changes to A, B, and C in the DoE should shift this distribution of T higher or lower, but the standard deviation should be fixed based on our baseline results.

At this point I’m not sure how to proceed. To keep things simple, we can assume that A, B, and C are linearly related to T. A naive thing to do would run a linear model on these DoE parameters to predict T (ex: modeled as a normal distribution with a fixed standard deviation from our baseline data), but this implicitly is assuming that all DoE experiments have the same mean when in reality the mean should be a function of the DoE parameters themselves. So really I should start out by (somehow) modeling how the T distribution mean shifts as a function the DoE parameters.

Thank you in advance for any help!

2

Update: here’s what I’ve come up with so far. I preprocessed my DoE or “study” (prefix: st_) data by standardizing all features st_feature_vals and targets st_target_vals (resulting in each field/column having mean 0 and variance 1) Using the same standard scalers, I then transformed the baseline (prefix: bl_) targets bl_target_vals.

I standardized the data this way to ensure that the linear regression (i.e. the last line in the code below) will be fit on standardized values, ensuring that the resulting coefficients are easily interpretable a la “feature importance” - please correct me if this is wrong!

One decision I made was to have beta as the linear coefficients of the delta relative to the baseline mean bl_mu. I also enforced the same bl_std for the study distribution.

with pm.Model() as model:
    # Set data
    X = pm.Data('X', st_features_vals)
    
    # Baseline distribution hyperparams
    bl_mu = pm.Normal('bl_mu', mu=bl_mean_val, sigma=5)
    bl_std = pm.Normal('bl_std', mu=bl_std_val, sigma=5)
    
    # Fit baseline target
    bl_y = pm.Normal('bl_y', mu=bl_mu, sigma=bl_std, observed=bl_target_vals)
    
    # Define coefficients for linear regression
    num_params = st_features_vals.shape[1]
    beta = pm.Normal('beta', mu=0, sigma=5, shape=num_params)
    
    # Fit with observations
    st_y = pm.Normal('st_y', mu=tt.dot(X, beta) + bl_mu, sigma=bl_std, observed=st_target_vals)  # fit with linear regression; including "+ bl_mu" forces beta to be fit to deltas as opposed to absolutes

If I’m interpreting my model correctly, the beta parameters model each input feature contribution to shifting the population mean relative to the baseline distribution - which is what I set out to do!

The last thing I wanted to check with the community is:

  • Am I neglecting/losing any information by not considering the baseline features at all? Would it be more complete to instead have X in the last line represent the entire baseline + study dataset, and the same for the observed. If so, how should I preprocess my features/targets to ensure my beta parameters are easily interpretable as feature importances?