Uncorrelated Covariance Matrix for an MvNormal

Questions
1

I’m working with a Multivariate Normal likelihood function. I seek to model observations as uncorrelated and therefore the covariance matrix should simplified to just the individual variable’s point wise variance along the diagonal.

I’d like to specify individual variable variance as hyper-priors with variable specific variances. How does one best construct a covariance matrix like this?

Here’s some example code for reference:

#Define Forward model (wrapped through theano)
with model:
    R, A, M = PDD_forward.forward(z_obs,
                                  f_s_prior,
                                  C_prior,
                                  f_r_prior,
                                  grad_a,
                                  A_m_prior)
    #Observation matrix   (Mx3)
    mu_obs  = np.array([R_obs, A_obs, M_obs]).T

    #"Prediction" matrix   (Mx3)
    mu_pred = tt.transpose(tt.stack([R, A, M]))

   #Unclear how to formulate covariance mat 
   # cov = ??? 

   #likelihood function 
   vals = pm.MvNormal('vals', mu=mu_pred, cov=cov, observed=mu_obs)

If my observation matrix is M \times 3, then what shape should the (uncorrelated) covariance matrix be?

For reference, I’m trying to mimic the likelihood function (Eqn. 13) from this paper (in my case tri-variate instead of b-variate) .

If I can provide any more info or clarify anything, please let me know!

Thanks in advance.

Best,
Andrew

1 Like
2

If you are going to assume that observations are independent, is there any reason to use a multivariate normal?

3

I guess not. In short I’d really just like to reproduce the likelihood function (Eqn. 13) from this paper. If that can be done without a multivariate normal then that’s fine by me.

Thanks!

4

Something like cov = tt.eye(3) * individual_variances? where individual variances is a vector of 3 random variables like Exponential(..., shape=3)

5

Thanks! I’ll try to implement this and report back.

I’m relatively new to both pymc3 and Bayesian methods in general. Could you elaborate on how the method above, something along the lines of:

#Define Forward model (wrapped through theano)
with model:
    R, A, M = PDD_forward.forward(z_obs,
                                  f_s_prior,
                                  C_prior,
                                  f_r_prior,
                                  grad_a,
                                  A_m_prior)
    #Observation matrix   (Mx3)
    mu_obs  = np.array([R_obs, A_obs, M_obs]).T

    #"Prediction" matrix   (Mx3)
    mu_pred = tt.transpose(tt.stack([R, A, M]))

   #Unclear how to formulate covariance mat 
   individual_variances = Exponential(..., shape=3)
   cov = tt.eye(3) * individual_variances

   #likelihood function 
   vals = pm.MvNormal('vals', mu=mu_pred, cov=cov, observed=mu_obs)

differs from something like:

with model:
    R, A, M = PDD_forward.forward(z_obs,
                                  f_s_prior,
                                  C_prior,
                                  f_r_prior,
                                  grad_a,
                                  A_m_prior)

    R_sigma = pm.HalfCauchy("R_sigma", ...)
    A_sigma = pm.HalfCauchy("A_sigma", ...)
    M_sigma = pm.HalfCauchy("M_sigma", ...)

    #Observation matrix   (Mx3)
    mu_obs  = np.array([R_obs, A_obs, M_obs]).T
    
    #"Prediction" matrix   (Mx3)
    mu_pred = tt.transpose(tt.stack([R, A, M]))

    #matrix of variances 
    sigmas  = tt.transpose(tt.stack([R_sigma, A_sigma, M_sigma]))

    #likelihood function 
    vals = pm.Normal("vals", mu= mu_pred, sigma=sigmas, observed=mu_obs)

The first code block uses a MvNormal for the likelihood, whereas the second just uses Normal but both seem to take an M\times3 matrix for mu and observed. From some early testing, they appear to produce similar results.

Appreciate the help!

6

A MvNormal with a multiple of the identity covariance (no covariance between dimensions) is indeed equivalent to a vector of Normals, except one is defined in terms of variances and the other in terms of standard deviations. So you can use whichever you prefer.

7

Awesome! Thanks for pointing out variance versus standard deviation difference, that slipped past me.

1 Like