Edit: See below for my attempt to build what I described below into a PyMC3 model. I also have a few conceptual questions listed at the bottom of the next post.
Hi everyone! I’m familiar with simple PyMC3 models (ex: coin flips, means comparisons, etc.) but I’m beginning to venture into the more powerful forms of interference.
Specifically, my team ran a design-of-experiment (DoE) matrix where we varied three parameters (
C) and measured the response (
T) in an effort to determine the effect of each parameter relative to our current baseline parameters. We ran quite a few baseline experiments in addition to the DoE matrix to build up data.
I’m trying to conceptually think through how I should set up the model in order to tease out these effects. My initial thought was to model the baseline results’
T distribution as a starting point, as the intention is to ultimately measure the effect of changing
C on this response. Modeling the baseline response alone has an added bonus of measuring the baseline-to-baseline variability (or really, variability inherent to what we’re testing) given fixed parameters.
To summarize up to here - our baseline experiments can be used to understand the variability (ex: standard deviation) between samples tested with our baseline parameters. The changes to
C in the DoE should shift this distribution of
T higher or lower, but the standard deviation should be fixed based on our baseline results.
At this point I’m not sure how to proceed. To keep things simple, we can assume that
C are linearly related to
T. A naive thing to do would run a linear model on these DoE parameters to predict
T (ex: modeled as a normal distribution with a fixed standard deviation from our baseline data), but this implicitly is assuming that all DoE experiments have the same mean when in reality the mean should be a function of the DoE parameters themselves. So really I should start out by (somehow) modeling how the
T distribution mean shifts as a function the DoE parameters.
Thank you in advance for any help!