Is it supposed to be like that?
No. If your rows are independent observations (so you don’t have some pesky MatrixNormal lying about) you can use Minibatch ADVI to speed it up significantly. Just change
pm.SomeDistribution(*args, observed=data)
to
batch=pm.Minibatch(data, 50) # or whatever size
pm.SomeDistribution(*args, observed=batch)
Or (perhaps even better) just run the initial step on a small random subset of the data.
Edit: Also, the default number of steps for ADVI is something ridiculous like 200,000. It tends to converge long before that, so I tend to tune it down significantly. There are callbacks that are supposed to stop it early, but for whatever reason I’ve never gotten them to work.