Bayesian IRT Pymc3 - Parameter inference

I would like to estimate IRT model using PyMC3.
I generated data with the following distribution:

alpha_fix = 4
beta_fix = 100
theta= np.random.normal(100,15,1000)
prob = np.exp(alpha_fix*(theta-beta_fix))/(1+np.exp(alpha_fix*(theta-beta_fix)))
prob_tt = tt._shared(prob)

Then I created a model using PyMC3 to infer the parameter:

irt = pm.Model()
with irt:
#     Priors
alpha = pm.Normal('alpha',mu = 4 , tau = 1)
beta = pm.Normal('beta',mu = 100 , tau = 15)
thau = pm.Normal('thau' ,mu = 100 , tau = 15)

#     Modelling
p = pm.Deterministic('p',tt.exp(alpha*(thau-beta))/(1+tt.exp(alpha*(thau-beta))))

out = pm.Normal('o',p,observed = prob_tt)

Then I infer through the model:

with irt: 
    mean_field = pm.fit(10000,method='advi', callbacks=[pm.callbacks.CheckParametersConvergence(diff='absolute')])

Finally, Sample from the model to get compute posterior:

pm.plot_posterior(mean_field.sample(1000), color='LightSeaGreen');

But the results of the “alpha” (mean of 2.2) is relatively far from the expected one (4) even though the prior on alpha was well-calibrated.

Would you have an idea of the origin of this gap and how to fix it?

Thanks a lot,

Hi

I cannot offer a full answer but I noticed that the sampling stops after 10,000 before the convergence call back kicks in. I increased the n iter to 100,000 and the convergence halts sampling after around 50k or 60k steps.

Now the alpha is closer to 3.4 with approx 25% of the mass above 4.0

1 Like