Rookie Question On Combining PCA and Bayesian Inference

Let’s just forget about the whole Laplace/PCA/shrinkage stuff altogether and make sure we’re on the same page with vanilla linear models. All Gaussian linear models follow the same form:

y \sim N(\mu, \sigma)
\mu = \alpha + \sum_{i=0}^k \beta_i \cdot X_i

For you, y is GDP, and you have either a bunch of economic time series or some principal components as X.

Either way, you need to put priors on the parameters in your model. If you have 1 principal component (let’s call it X), then your model is:

\text{GDP}_t \sim N(\mu_t, \sigma)
\mu_t = \alpha + \beta \cdot X_t

So there are 3 parameters: \alpha, \beta, and \sigma. Using my language from before, k=1 because there is one covariate, X. You need to tell the model what you think reasonable values for these are by setting priors. This will, in turn, determine your beliefs about the possible values of GDP.

If you use more things to model \mu_t, you will have more parameters. For example, with 3 factors, call them C_t, I_t, L_t for consumption, investment, and aggregate hours worked, you will have:
\text{GDP}_t \sim N(\mu_t, \sigma)
\mu_t = \alpha + \beta_C \cdot C_t + \beta_I \cdot I_t + \beta_L \cdot L_t

Now you need priors on 5 parameters: \alpha, \sigma, and 3 betas. Using my language from before, k=3 now.

Whether you use PCA or not, none of this part changes.

The next question is what you’re actually trying to achieve, which seems to be variable selection. You have k=100, and you want to pick out a subset of only the most important variables. You could use PCA to get the first k components, which would be a big mix of everything. Or, you could use skeptical priors, such Laplace or “Spike and Slab”, to bias the values of \beta_i towards zero, removing them from \mu_t.

I made a small gist here with a couple examples that might be useful to you, but I strongly recommend you go over the Statistical Rethinking course to build an understanding of what goes on in these models.

1 Like