Using Bayesian for estimating parameter

Questions
4

I think there is a lot of subtle naming issues with your example. I’ll try to explain the basic process of writing down a model to infer some variables’ posterior distribution given some observed values. I still strongly recommend that you read the entire getting started guide because all the process that I’ll go through is also explained there.

The first step to get any posterior inference set up is to specify the statistical model that we assume underlies the data generation process for the thing that we will observe. Mathematically, what this means is that we will try to write down the joint probability that a given observation occurs (the likelihood) as a function of some parameters that we may know with absolute certainty, or that we don’t know and thus represent as unobserved random variables. This latter case means that we must assume that a priori that these uncertain parameters must come from some other probability distribution. This means that the statistical model will have three main parts:

  1. Some variables that we will be able to observe somehow, for which we write the likelihood as a function of some parameters
  2. Parameters for which we don’t know with absolute certainty their values, so we assume a priori that they are sampled from some probability distribution (which can have parameters itself, leading to hierarchical statistical models)
  3. Parameters whose values we know with absolute certainty (so there is no a priori probability distribution associated with them).

I’m sorry if this is known to you but I felt that there was some confusion in your post because you said that your observations come from a normal with some parameters values that you already knew, and then there was a uniform prior, which was not associated to any parameter of your observed data. The terminology can be a bit confusing but it is very important.

Ok, we went through some really general stuff and now we’ll see how the statistical model is specified in pymc3 and then how you infer the posterior distribution.

First I’ll write down how a particular model would look like with just mathematical notation:
obs_{\mu} \sim N(\mu=1.3, \sigma=0.05)
obs_{\sigma} \sim U(\mathrm{lower}=0, \mathrm{upper}=1)
obs \sim N(\mu=obs_{\mu}, \sigma=obs_{\sigma})

The statistical model that I wrote down represents the process we assume underlies the generation of observations obs. I assume that we don’t know a priori with full certainty neither the mean nor the standard deviation of the observations. I also assume a prior distribution for the observation’s mean and standard deviation parameters (a normal for obs_{\mu} with 1.3 mean and 0.05 deviation, and a uniform between 0 and 1 for obs_{\sigma}).

This model would be written in pymc3 as

import numpy as np
import pymc3 as pm

observed_values = np.random.randn(100) # Here you would put your actual observations
with pm.Model() as model:
    obs_mu = pm.Normal('obs_mu', mu=1.3, sigma=0.05)
    obs_sigma = pm.Uniform('obs_sigma', lower=0, upper=1)
    obs = pm.Normal('obs', mu=obs_mu, sigma=obs_sigma,
                    observed=observed_values)

Note that it is really easy to write down the statistical model with the conditional dependencies between the a priori unknown parameters and the observed data. Now we’ll draw samples from the posterior distribution like this:

with model:
    trace = pm.samples(2000)

The trace contains samples of your unobserved a priori unknown random variables that match their posterior probability distribution given the observed data. You can view the trace in many ways but I’ll show the two simplest ones (refer to arviz for a detailed list)

with model:
    pm.plot_posterior(trace)
    pm.traceplot(trace)

The first plots the marginal posterior distributions for each of the unknown parameters (for the joint probability you have to look at plot_pair. The second shows the trace of the markov chain for each parameter.

I’m sorry if I still haven’t understood your particular statistical model because you seem to already know all the observation’s distribution parameter values, so I don’t know to which parameter the prior corresponded to. At least I hope that I was detailed enough to help you express your statistical model more clearly.

1 Like