Having Trouble translating Winbugs code

Thanks. I will experiment some more.

in my first experiment I took out one deterministic variable at time got rid of most of them. I ran the model and it produced the estimates below. The mean values are similar,

image

I put the deterministic variables back in and got what I think are cleaner results. the r-hat and essential sample size statistics took a hit when got rid of the deterministic variable. Maybe it’s just one item that is causing the problem. For the models I am running 100,000 samples on 4 cores and using the nutpie sampler.

image

Here is the code again. When I did not wrap p1, p2, and p3 as deterministic variables, the number of divergences increased. Is there any advantage in terms of model accuracy in minimizing the number of deterministic variables? I would like to get rid of the clutter so long as it does not hurt the estimates. In winbugs, my impression is that you declare the variables, and then ask which ones you want to trace, and whether you want to trace them or not does not impact the model.

with pm.Model(coords=coords) as model:

    # set priors
    d_=pm.Normal("d_",mu=0,tau=0.0001,shape=(6,))
    d=pm.Deterministic("d",pm.math.concatenate([[pt.constant(0)],d_]),dims="treatments")

    sd=pm.Uniform("sd",0,5)
    tau=pm.Deterministic("tau",1/(sd*sd))    

    # Set mu (same for all sets of comparisons)
    mu=pm.Normal("mu",mu=0,tau=0.0001,dims="studies",shape=(N,1))

    # Arm 1    
    r1=pm.ConstantData('r1',R1)
    n1=pm.ConstantData('n1',N1)
    t1=pm.ConstantData('t1',T1)    
    p1=pm.Deterministic("p1",pm.invlogit(mu))
    y1=pm.Binomial("y1",n=n1,p=p1,observed=r1)
    
    # Arm 2
    r2=pm.ConstantData('r2',R2)
    n2=pm.ConstantData('n2',N2)
    t2=pm.ConstantData('t2',T2)
    md2=pm.Deterministic("md2",d[t2]-d[t1])
    taud2=pm.Deterministic("taud2",pm.math.zeros_like(mu)+tau)
    delta2=pm.Normal("delta2",mu=md2,tau=taud2)         
    p2=pm.Deterministic("p2",pm.invlogit(mu+delta2))
    y2=pm.Binomial("y2",n=n2,p=p2,observed=r2)
    w2=pm.Deterministic("w2",delta2-d[t2]+d[t1])
    
    # Arm 3
    r3=pm.ConstantData('r3',R3)
    n3=pm.ConstantData('n3',N3)
    t3=pm.ConstantData('t3',T3)
    s3=pm.ConstantData('s3',S3)
    k3=pt.constant(3)
    sw3=pm.Deterministic("sw3",(w2[s3])/(k3-1))
    taud3=pm.Deterministic("taud3",pm.math.zeros_like(t3)+tau*2*(k3-1)/k3)
    md3=pm.Deterministic("md3",d[t3]-d[t1[s3]]+sw3)
    delta3=pm.Normal("delta3",mu=md3,tau=taud3);
    p3=pm.Deterministic("p3",pm.invlogit(mu[s3]+delta3))
    y3=pm.Binomial("y3",n=n3,p=p3,observed=r3)

    redata=pm.sample(100000,tune=10000, nuts_sampler="nutpie")

pm.Deterministic does absolutely nothing computational. If you get different results when you remove them, it’s because you’re inadvertently changing something else about the model.

On another note, your number of tune and draws is very high. Did you need them because the model was not converging well?

99 out of 100 times that means the model may need some more work, as it’s badly parametrized or there’s a subtle bug.

It may also mean the changes you’re seeing are not Deterministic vs not (which is bery unlikely) but maybe just the random seed changing for a model that’s very finicky

I think that the model is challenging because of the number of parameters that I’m asking it to estimate. The only reason that I think it is correct (or almost correct) is that I am close to the winbugs output that I posted above. The winbugs code has been used in several studies so it has been subject to peer review, so I hope it is correct. Again, I am asking a small amount of data to estimate 6 differences between treatments, which is the focus of the analysis, in addition to 36 study mean values, and 37 random treatment effects. Of course, there may also be a subtle bug that I’m missing. I already found several that I corrected.

I am running 400,000 samples with 40,000 tunning samples. Due to randomness, every time I run the model I get a different number of divergencies, in 5 models runs the number ranged from 16,226 to 31,514. The mean values look similar in each run. The first place I will look for a bug is in the multi-arm correction code. Thank you for the reminder about using the same random seed when I do my testing. I used my favorite seed (8675309) and it produced 23,976 divergencies two times in a row, which is around the average when I was not setting the seed.

If (or when) the model is correct but is challenging for the sampler I’m not sure what remedial action to take to reduce the number of divergences to clean up the distribution plots and the distributions statistics.

The estimates of the analysis will be used in a simulation model. It is not for a web application where a user clicks a button and then expects an answer in a few seconds. Getting an answer quickly is good, but not critical.

Your model is quite small. It should not be an issue to sample, provided it’s well specific and correctly implemented. Getting any divergences suggest this is not the case.

Translating models from Gibbs frameworks like winBUGS to HMC often requires more than one-to-one translation. One reason is that when writing models for use with a Gibbs sampler, the authors were constrained to choose certain priors or parameterizations that are compatible with the sampler. These might be really bad choices for HMC-family samplers. The uniform prior on the standard deviation is one such example I already highlighted. Another issue is that HMC samplers provide a lot of diagnostics about the sample process, including divergences. All Gibbs (or Metropolis, or any non-gradient sampler) can provide for diagnostic is convergence checks between chains. But these are local diagnostics, in the sense that they tell you only something about where the sampler ended up exploring. They say nothing about whether the sampler was able to explore the entire sample space, or whether there were regions (like funnels or corners) where it couldn’t get into. This is exactly what divergences are telling you about. Ignoring them leads to biased inference. See this lecture by Michael Betancourt for a discussion.

1 Like

I got rid of most the deterministic variables. I set the random seed to the same number each time I removed a deterministic wrapper, and then looked at the number of divergences to see if the number changed, which it didn’t. Not setting the seed was providing misleading feedback.

Here is the new code


with pm.Model(coords=coords) as remodel:

    # set priors
    d_=pm.Normal("d_",mu=0,tau=0.0001,shape=(6,))
    d=pm.Deterministic("d",pm.math.concatenate([[pt.constant(0)],d_]),dims="treatments")

    sd=pm.Uniform("sd",0,5)
    tau=1/(sd*sd)

    # Set mu (same for all sets of comparisons)
    mu=pm.Normal("mu",mu=0,tau=0.0001,dims="studies",shape=(N,1))

    # Arm 1    
    r1=pm.ConstantData('r1',R1)
    n1=pm.ConstantData('n1',N1)
    t1=pm.ConstantData('t1',T1)    
    p1=pm.invlogit(mu)
    y1=pm.Binomial("y1",n=n1,p=p1,observed=r1)
    
    # Arm 2
    r2=pm.ConstantData('r2',R2)
    n2=pm.ConstantData('n2',N2)
    t2=pm.ConstantData('t2',T2)
    md2=d[t2]-d[t1]
    taud2=pm.math.zeros_like(mu)+tau
    delta2=pm.Normal("delta2",mu=md2,tau=taud2)         
    p2=pm.invlogit(mu+delta2)
    y2=pm.Binomial("y2",n=n2,p=p2,observed=r2)
    w2=delta2-d[t2]+d[t1]
    
    # Arm 3
    r3=pm.ConstantData('r3',R3)
    n3=pm.ConstantData('n3',N3)
    t3=pm.ConstantData('t3',T3)
    s3=pm.ConstantData('s3',S3)
    k3=pt.constant(3)
    sw3=(w2[s3])/(3-1)
    taud3=pm.math.zeros_like(t3)+tau*2*(3-1)/3
    md3=d[t3]-d[t1[s3]]+sw3
    delta3=pm.Normal("delta3",mu=md3,tau=taud3);
    p3=pm.invlogit(mu[s3]+delta3)
    y3=pm.Binomial("y3",n=n3,p=p3,observed=r3)

    redata=pm.sample(100000,tune=10000, nuts_sampler="nutpie", random_seed=8675309)

Thank you for the feedback on use of deterministic variables. I was able to see they were not needed when I used the same seed value. However, they were helpful when I was comparing the variables with those produced by wingbugs. The numbers are not the same because of random sampling but when something is very wrong it is clear when you compare the statistical mean values.

Winbugs also struggles with this model. When you look at the trace it struggles to stay on track, and has a lot of autocorrelation in the estimated parameters.

Do you think it would be worthwhile to have an example page for a model like this? The model was first developed around 2010 and then modified in 2016. It is possible that a newer model exists that takes advantage of people becoming more sophisticated using Bayesian models with more up to date software.

That question is easier to answer after such a model is found. The process can always be documented but whether it’s useful enough to warrant inclusion (and then maintenance) in the official docs is a bit too early to say

Here is one more example. On the first example there was only one study with three arms. This one has 4 studies with three arms and the estimates are a lot cleaner. In the example there is a variable for time to adjust for differential follow-up, but I need a inverse complimentary log log link to implement. Ignoring time, the output produces the plots below that look pretty clean.

Code below:

import arviz as az
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import pymc as pm
from pytensor import tensor as pt
from pytensor.ifelse import ifelse
import pytensor 

%matplotlib inline

import pandas as pd
import io
data_string="""studyid,time,t1,t2,t3,r1,r2,r3,n1,n2,n3,study
0,5.8,1,2,3,43,34,37,1081,2213,1102,1.MRC-E
1,4.7,1,2,,29,20,,416,424,,2.EWPH
3,3,1,2,,140,118,,1631,1578,,3.SHEP
4,3.8,1,3,,75,86,,3272,3297,,4.HAPPHY
5,4,1,4,5,302,154,119,6766,3954,4096,5.ALLHAT
6,3,1,4,,176,136,,2511,2508,,6.INSIGHT
7,4.1,1,5,,200,138,,2826,2800,,7.ANBP-2
8,1,1,6,,8,1,,196,196,,8.ALPINE
9,3.3,2,4,,154,177,,4870,4841,,9.FEVER
10,3,2,5,,489,449,,2646,2623,,10.DREAM
11,4.5,2,5,,155,102,,2883,2837,,11.HOPE
12,4.8,2,5,,399,335,,3472,3432,,12.PEACE
13,3.1,2,6,,202,163,,2721,2715,,13.CHARM
14,3.7,2,6,,115,93,,2175,2167,,14.SCOPE
15,3.8,3,4,5,70,32,45,405,202,410,15.AASK
16,4,3,4,5,97,95,93,1960,1965,1970,16.STOP-2
17,5.5,3,4,,799,567,,7040,7072,,17.ASCOT
18,4.5,3,4,,251,216,,5059,5095,,18.NORDIL
19,4,3,4,,665,569,,8078,8098,,19.INVEST
20,6.1,3,5,,380,337,,5230,5183,,20.CAPPP
21,4.8,3,6,,320,242,,3979,4020,,21.LIFE
22,4.2,4,6,,845,690,,5074,5087,,22.VALUE"""
data=pd.read_csv(io.StringIO(data_string),sep=",")

N=len(data)

N1=data[["n1"]].astype(dtype=np.int32).to_numpy()
N2=data[["n2"]].astype(dtype=np.int32).to_numpy()
N3=data[["n3"]].dropna().astype(dtype=np.int32).to_numpy()

R1=data[["r1"]].astype(dtype=np.int32).to_numpy()
R2=data[["r2"]].astype(dtype=np.int32).to_numpy()
R3=data[["r3"]].dropna().astype(dtype=np.int32).to_numpy()

T1=data[["t1"]].astype(dtype=np.int32).to_numpy()-1
T2=data[["t2"]].astype(dtype=np.int32).to_numpy()-1
T3=data[["t3"]].dropna().astype(dtype=np.int32).to_numpy()-1

TIME=data["time"].to_numpy()

S3=data.dropna(subset=["t3"])["studyid"]

K=np.array([1,2,3])

STUDIES=data["study"]
TREATMENTS=["Diuretic","Placebo","Beta Blocker","CCB","ACE inhibitor","ARB"]
coords = {"studies": STUDIES, "treatments":TREATMENTS}


with pm.Model(coords=coords) as remodel:

    # set priors
    d_=pm.Normal("d_",mu=0,tau=0.0001,shape=(5,))
    d=pm.Deterministic("d",pm.math.concatenate([[pt.constant(0)],d_]),dims="treatments")

    sd=pm.Uniform("sd",0,5)
    tau=1/(sd*sd)

    # Set mu (same for all sets of comparisons)
    mu=pm.Normal("mu",mu=0,tau=0.0001,dims="studies",shape=(N,1))
    
    # Arm 1    
    r1=pm.ConstantData('r1',R1)
    n1=pm.ConstantData('n1',N1)
    t1=pm.ConstantData('t1',T1)    
    p1=pm.invlogit(mu)
    y1=pm.Binomial("y1",n=n1,p=p1,observed=r1)
    
    # Arm 2
    r2=pm.ConstantData('r2',R2)
    n2=pm.ConstantData('n2',N2)
    t2=pm.ConstantData('t2',T2)
    md2=d[t2]-d[t1]
    taud2=pm.math.zeros_like(mu)+tau
    delta2=pm.Normal("delta2",mu=md2,tau=taud2)         
    p2=pm.invlogit(mu+delta2)
    y2=pm.Binomial("y2",n=n2,p=p2,observed=r2)
    w2=delta2-d[t2]+d[t1]
    
    # Arm 3
    r3=pm.ConstantData('r3',R3)
    n3=pm.ConstantData('n3',N3)
    t3=pm.ConstantData('t3',T3)
    s3=pm.ConstantData('s3',S3)
    k3=pt.constant(3)
    sw3=(w2[s3])/(3-1)
    taud3=pm.math.zeros_like(t3)+tau*2*(3-1)/3
    md3=d[t3]-d[t1[s3]]+sw3
    delta3=pm.Normal("delta3",mu=md3,tau=taud3);
    p3=pm.invlogit(mu[s3]+delta3)
    y3=pm.Binomial("y3",n=n3,p=p3,observed=r3)

    redata=pm.sample(100000,tune=10000, nuts_sampler="nutpie", random_seed=8675309)
	
az.plot_trace(redata,var_names=["d","sd"])

az.summary(redata,var_names=["d","sd"], hdi_prob=0.95)

Below is the Bambi code the the inverse of the complimentary log log link



def cloglog(mu):
    """Cloglog function that ensures the input is greater than 0."""
    mu = force_greater_than_zero(mu)
    return np.log(-np.log(1 - mu))


def invcloglog(eta):
    """Inverse of the cloglog function that ensures result is in (0, 1)"""
    result = 1 - np.exp(-np.exp(eta))
    return force_within_unit_interval(result)


def force_within_unit_interval(x):
    """Make sure data in unit interval is in (0, 1)"""
    eps = np.finfo(float).eps
    x[x == 0] = eps
    x[x == 1] = 1 - eps
    return x


def force_greater_than_zero(x):
    """Make sure data in positive reals is in (0, infty)"""
    eps = np.finfo(float).eps
    x[x == 0] = eps
    return x

Looks like this works. I get the same answer as in Program 3(a): Binomial likelihood, cloglog link, Random Effects (Diabetes example).

https://www.sheffield.ac.uk/media/34176/download?attachment

# New data

time=pm.ConstantData('time',TIME)

# Modified code 

p1=1-pm.math.exp(-pm.math.exp(pm.math.log(time)+mu))
p2=1-pm.math.exp(-pm.math.exp(pm.math.log(time)+mu+delta2))
p3=1-pm.math.exp(-pm.math.exp(pm.math.log(time[s3])+mu[s3]+delta3))
1 Like

I hope that an example page is set up. I would like to use PYMC for network meta analysis, but I don’t see any examples of it’s use online. Stan seem more likely to be used. I just ran across PYMC and I like the way it works and integrates into python. I will see if I can use it in a project with NMA. It would be easier to convince people, including regulators, if I could point to journal articles and a peer reviewed example page on your site.

I found coding at first a little difficult because in both winbugs and PYMC you make declarative statements and then the model is implemented with code that builds a computational graph. For me, it was not clear what was happening, so it made it difficult to implement something that was slightly different than the other examples. I think my code works now. I did another model with time and was able to use a complementary log-log link to adjust for differential follow-up times, and match the published results. Also, the summary statistics and distribution plots and traces look good.

Thank you for your comments and advice. I was looking for some help for avoiding doing things that don’t make any sense. Your advice on not overusing deterministic variables was the type of advice I was looking for.

To understand what happens under the hood, this may help: PyMC and PyTensor — PyMC 5.13.1 documentation

You are basically building the computational graph in real time, instead of it being parsed from a text file like it is in bugs/stan. The nice thing is you can also introspect it / parts of it yourself

Thanks. I looked at the site earlier and tried the examples. I also found a youtube video that you did to speed up matrix multiplication. Both are very good. However, to me it mostly let me know that things were happening that I didn’t understand and was not in control of. I decided to plow my way through and hope that things would work out. So far so good.

Being able to inspect using “eval” was helpful in getting the shapes right. With the winbugs code they also do a lot of things in the model probably because it his harder to get to the samples out. With PYMC much of this code could be run after the basic model is estimated.

1 Like