Implementing Logistic Regression with OVR using PyMC3

Questions
1

I’m trying to implement a logistic regression model with One vs Rest(OvR) stratergy for multi-class classificaiton using PyMC3. I found the softmax regression code from this git-hub issue thread and tried to change it to a OvR model. This is my code.

with pm.Model() as self.logistic_model:
       #feature count is X.shape[1]
       alpha = pm.Normal('alpha', mu=0, sd=20, shape=class_count)
       beta = pm.Normal('beta', mu=0, sd=20, shape=(feature_count, class_count))
       
       #link the RV with target variable (y)
       mu = alpha[y] + tt.dot(X, beta[y])
       # sigmoid function instead of softmax
       p = pm.Deterministic('p', tt.nnet.sigmoid(mu))

       y = pm.Categorical('y', p=p, observed=y)
       trace = pm.sample(1000, step=pm.Metropolis())

As shown by the hierarchical linear regression model tutorial, I tried to link the target variable (y) and random variables. But, after that change the dot product between X and beta fails due to shape mismatch.

I can’t figure out a way to sample separate random variable from the random matrices for the each class. Any help, or suggestion will be appreciated.

Thanks

2

One vs Rest usually involves training multiple classifiers separately, which would means you build n logistic regression models.

But I guess in this case you can do it all in one model:

with pm.Model() as model:
    alpha = pm.Normal('alpha', mu=0, sd=20, shape=class_count)
    beta = pm.Normal('beta', mu=0, sd=20, shape=(feature_count, class_count))
    
    mu = alpha + tt.dot(X, beta)
    for i in range(class_count):
        # sigmoid function instead of softmax
        p = pm.Deterministic('p%i'%i, tt.nnet.sigmoid(mu[:, i]))
        pm.Bernoulli('y%i'%i, p=p, observed=(y==i))
1 Like
3

Thanks junpenglao, this is exactly what I wanted (to have all classifiers in the same model rather than initiating multiple pymc3 models).