Weighted Rank Aggregation

Questions
1

I have been working on some weighted rank aggregation task that I believe could be much improved by leveraging pymc3. The goal is to take a set of ordinal rankings of K entities by J judges and infer an aggregate ranking of the K entities which minimize pairwise disagreements with the input judges rankings. I have previously used the Kendall Tau distance as a metric for this. Ideally, this would also take into account a weight based on the confidence in each judge.

I have played around with the Thurstonian model included in the github examples: https://github.com/pymc-devs/pymc3/blob/master/pymc3/examples/rankdata_ordered.py

I was wondering if someone has tried to implement something similar or might be able to shed some light on how to expand that example model.

  1. In the example, there is a comment indicating that “using sd does not work yet”. Why is this the case. Is this an underlying limitation of pymc3 or the implementation in the example?

  2. Is it possible to model each K ranker as a SkewNormal instead of a standard deviation? All of my initial attempts proved unsuccessful in getting pymc3 to model the alpha parameter using something like “mu_hat = pm.SkewNormal(“mu_hat”, mu=0, sigma=1, alpha=2, shape=K - 1)”.

  3. Is it possible to extend this example to include weights or confidence in the input rankings (rows of y_argsort)?

Thank you!