Some fun with GitHub Copilot and PyMC

I just got access to GitHub Copilot and thought it would be fun to see whether or not their language model can capture the subtleties of Bayesian modeling. I loaded it up in VSCode and gave it a few prompts. Here’s what I found:

``````import pymc3 as pm

def pymc_regression_model(X,y):
``````

And here’s what it suggested:

``````def pymc_regression_model(X,y):
with pm.Model() as model:
alpha = pm.Normal('alpha', mu=0, sd=10)
beta = pm.Normal('beta', mu=0, sd=10, shape=X.shape[1])
mu = alpha + pm.math.dot(X, beta)
pm.Normal('y', mu=mu, sd=1, observed=y)
return model
``````

which looks like a perfectly reasonable linear regression. The only thing I would pick on is that the error variance is fixed to 1.

Let’s try a logistic regression version, by modifying the above function definition to `pymc_logistic_regression_model` instead. This is what we get:

``````with pm.Model() as model:
# Define the priors
alpha = pm.Normal('alpha', mu=0, sd=10)
beta = pm.Normal('beta', mu=0, sd=10, shape=X.shape[1])
# Define the likelihood
likelihood = pm.math.dot(X, beta) + alpha
pm.glm.GLM.from_formula('y ~ x', data=dict(x=X, y=y), family=pm.glm.families.Binomial())
# Inference!
step = pm.Metropolis()
trace = pm.sample(100000, step=step)
return trace
``````

It looks like it’s struggling to get the right format. It has elements of the right answer, i.e. using the GLM module with a non-Gaussian likelihood, but it selects the binomial link instead. It also looks like it’s attempting to use both the GLM module and the standard model definition via priors and likelihoods, which isn’t a good idea. It also tries to use the Metropolis-Hastings sampler >:( rather than relying on NUTS, which wouldn’t make sense for a logistic regression model.

I tried many other prompts including `pymc_spatial_regression`, `pymc_spatial_logistic_regression` and `gaussian_process_model` but these more elaborate prompts all returned essentially a basic linear regression (though with docstrings that mentioned spatial / GP parts). It did a little better with `pymc_ab_test` which at least used a Bernoulli likelihood which would make sense for click-through and other types of discrete statistics.

4 Likes

I appreciate such enthusiasm in my bot collaborators (though it seems it may be parroting Thomas’ enthusiasm?).

2 Likes

Do we know how old is the code it was trained on? I feel like it is missing using the map as starting point to be a return to the past .

The main takeaway is probably that even for super simple models it can steal from other people verbatim (which is the only case I see copilot returning any model that makes a little bit of sense) it will not fare well at all with v4, . We have accepted both `sd` and `sigma` for quite a long time, with `sd` not being documented for a couple years at least. `sd` is gone for good in v4. The glm module is also gone.

3 Likes