I’m new to using PYMC/Bayesian modeling, so as a little test drive, I decided to record the item levels of players in world of warcraft before and after the start of this latest season. I was interested in seeing how the mean item level evolves overtime so I am going to record data every week for at least 5 weeks of the season.
I have data of the pre-season week (week 0) and the first week (week 1).
I ran around in-game and collected 100 player’s worth of item levels for each week. I then used a Normal and StudentT likelihood function with relatively flat priors for the parameters.
I have the below plots from the posterior predictive. It looks like the StudentT is a better model as it has a sharper peak in the center. Is there a way to tell more than just visually if the StudentT is a better choice over the Normal?
The 94% HDI for the means of week 1 and 0 are [614,620], [620,626] for the normal model
and [615,621] , [623,629] for the StudentT.
Cool project! I did molten core raids back in college rather than going to statistics class. We were second on the server to craft Sulfuras. Couldn’t even get first, cringe.
The difference between a Normal and StudentT lies entirely in the behavior of the tails, not the body. It’s good to think about the choice in terms of outliers. The assumption of the Normal is that around 98% of the data falls inside the range \mu \pm 2\sigma. You didn’t post the estimated parameters, but lets say it’s \mu = 625 and \sigma = 25. So your model says basically everyone should be between 575-675, which is more-or-less what we see in the data.
There is one weirdo who was below 575 in week 0: how should the model react to him? The normal will either shift the mean left somewhat or increase the variance somewhat in order to explain his presence.
The Student T distribution has another option: it can decrease \nu, which controls the thickness of the tails. So the mean and variance estimates can stay the same, but you can accommodate unusually high or unusually low observations.
So the modeling choice comes down to your belief about the nature of the population of WoW players. To oversimplify a lot, if you think everyone is more or less the same, you pick a normal. If, on the other hand, you think that in addition to a core population that behaves more or less the same, there are also sub-populations that behave radically different – gearing up much faster or slower than the core – then a Student T is appropriate.
I think there is a major split between players, “active/main character” and “new, returning, or alts”
The long tails on the lower end are caused by the split. To remedy this maybe I should have rejected data below the “easiest” obtainable item level for someone who actively play which would have removed data below 580.
On the higher end I don’t think there’s too much of a concern as there is a maximum item level that you can achieve no matter how much effort you put in. Last season it was 639, this season it is 679. You can see how there’s a bump in week0 at the top end as players who could have pushed above 639(based on skill) all got grouped together at the maximum.
If there’s (at least) two groups – but you don’t know which data comes from which group – you could also try a mixture model, and assign each datapoint to a latent class (that you would interpret as main vs alt). Each group would have it’s own mean and variance that you could learn separately. As a bonus, you would get an estimate of how likely it was that a given observation was a main.
