# Modeling reinforcement learning of human participant using PyMC3

**URL:** <https://discourse.pymc.io/t/modeling-reinforcement-learning-of-human-participant-using-pymc3/1735>\
**Category:** Questions\
**Created:** [August 14, 2018, 10:55pm UTC](https://discourse.pymc.io/t/modeling-reinforcement-learning-of-human-participant-using-pymc3/1735 "2018-08-14T22:55:22Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![Maria](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.pymc.io/maria/32/1038_2.png) [@Maria](https://discourse.pymc.io/u/Maria)\
**Post date:** [August 30, 2018, 8:06pm UTC](https://discourse.pymc.io/t/modeling-reinforcement-learning-of-human-participant-using-pymc3/1735/3 "2018-08-30T20:06:25Z")

</div>

Hi Junpeng,

Thank you for your thoughts!

I chose better priors for `beta`, and fixed the ranges for `alpha` and `forget`, like you suggested. I was not sure how to use the softmax instead of `T.repeat` for the weights though, because my matrix is 4d and won’t be accepted by `T.nnet.sofmax` - or were you thinking about a different softmax function? As for the scan, I don’t think there is a way to get rid of it unfortunately, because each row of `Q_sub` depends on the previous row, so I need to pass the previous results to new each iteration.

For the priors, I am now using:

```auto
beta_mu = pm.Gamma('beta_mu', mu=1, sd=2, testval=1.25)
beta_sd = pm.HalfNormal('beta_sd', sd=0.5, testval=0.1)

alpha_mu = pm.Uniform('alpha_mu', lower=0, upper=1, testval=0.15)
alpha_sd = pm.HalfNormal('alpha_sd', sd=0.1, testval=0.05)

forget_mu = pm.Uniform('forget_mu', lower=0, upper=1, testval=0.06)
forget_sd = pm.HalfNormal('forget_sd', sd=0.1, testval=0.03)

```

I fixed the ranges of all parameters using `pm.Bound` on the individual non-central parameters:

```auto
beta_matt = pm.Bound(pm.Normal, lower=-beta_mu / beta_sd)(
        'beta_matt', mu=0, sd=1, shape=n_subj, testval=np.random.choice([-1, 0, 1], n_subj))
beta = pm.Deterministic('beta', beta_mu + beta_sd * beta_matt)
    
alpha_matt = pm.Bound(pm.Normal, lower=-alpha_mu / alpha_sd, upper=(1 - alpha_mu) / alpha_sd)(
        'alpha_matt', mu=0, sd=1, shape=n_subj, testval=np.random.choice([-1, 0, 1], n_subj))
alpha = pm.Deterministic('alpha', alpha_mu + alpha_sd * alpha_matt)

forget_matt = pm.Bound(pm.Normal, lower=-forget_mu / forget_sd, upper=(1 - forget_mu) / forget_sd)(
        'forget_matt', mu=0, sd=1, shape=n_subj, testval=np.random.choice([-1, 0, 1], n_subj))
forget = pm.Deterministic('forget', forget_mu + forget_sd * forget_matt)

```

After these fixes, the results still look the same (excuse the small number of samples).  
First, flat model:

 ![flat_abf_fixed_priors_2018_8_29_10_38_humans_n_samples100alienstrace_plot](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/b/b0418705285744f439b37e97473b352d1e983f59.jpeg)  
Second, hierarchical model:  
 ![hier_abf_fixed_priors_2018_8_29_10_39_humans_n_samples100alienstrace_plot](https://canada1.discourse-cdn.com/flex036/uploads/pymc3/original/2X/a/a20d0c980f47f85179079a5e797d9a847ea5a661.png)

I am nor sure what to try next. Do you have any ideas?

I also found a [great PyMC3 tutorial](https://github.com/fonnesbeck/PyMC3_EUSS) by Chris Fonnesbeck with a notebook about model checking and will go through it next, to see what else I could check or fix. Do you know of any other resources that could be helpful?

Thanks!

---

_[View the full topic](https://discourse.pymc.io/t/modeling-reinforcement-learning-of-human-participant-using-pymc3/1735)._