Advice on dimensionality reduction?

In practice, if you can subset the rows of your data and still pass it into your model (without, of course, changing the model), then you can use minibatch ADVI.

The cases where you can’t do this are typically because you’re modeling covariances between rows as well as between columns.

For time series, the typical approach is to introduce lag variables (thereby increasing the number of columns); and then the model can utilize the lags within the same row rather than pulling values from previous rows. This then makes each row conditionally independent (given the lag variables) and thus amenable to sub-sampling, bootstrapping, and minibatch fitting.