Random intercepts with high-cardinal categorical feature?


Any suggestions for estimating a binary outcome with a multi-level model with random intercepts on a categorical feature that could potentially have millions of distinct items i.e is highly cardinal?

For example - if I wanted to build a multi-level model for an IoT network that has device ID and some outcome of interest, I might want to have a random intercept per device ID. Doable with PyMC3? Doable… at all?

1 Like

Interesting question and I’m interested to hear other’s thoughts too.

It’s certainly possible to specify models with high dimensionality - you just have to wait longer - but it might help to consider your use-case a little more before diving in. I could see it being useful to set a baseline for a device in the case that you take repeated measurements over time - is that your use-case and if so do you expect the variance between devices to be substantial?


The use case is optimizing decisions for each device. For example, if we want to create some kind of send time optimization process to e.g. deliver push notifications at the time a user is mostly to be engaging with their device, we would want to estimate the most likely time of engagement for a particular user. For users with a lot of engagement history, we would want that estimate to lean more on their own engagement. For users with less (or no) engagement history, we would want to shrink the estimate towards the average. This way we can get progressively more personalized estimates as users engage with their device more while maintaining our “best guess” for those that have little or data.

Ah I see - you certainly could try - though it sounds like you might end up with an interaction of user and time-of-day, which would explode the parameter space. There’s likely periodicity in the data which could be used in interesting ways… Strictly my ill-informed 2p, but if it were me I’d probably try to conjure some new features and fit a pooled model first.

Yeah, I think complete pooling is going to end up being the starting approach until we can figure out how to do a partial pool at scale. Thanks for your opinion.

1 Like