Scalable Bayesian Structural Time Series

,
1

Hi (@jessegrabowski @Dekermanjian @ChrisFonnesbeck ,

I’m Surjeet from Bengaluru. I work on the production trading desk at WMX where we run real-time Bayesian state-space models daily for gold futures, BTCUSD, and forex. The significant issue we hit is having to re-run full MCMC every time new ticks arrive which can be solved by this Scalable Online Bayesian State Space Model.

I’ve been following the recent discussion here about online_step(), marginalized Kalman Filter, Van Loan discretization, and storage options (zarr vs netcdf). This stuff is super relevant to live trading pipelines.

I’m putting together a proposal for the online SSM project and looking for my PR getting merged.
Would really appreciate any quick feedback from the mentors before the deadline. Happy to share more about our real trading use-cases if it helps.

Thanks!
Surjeet (GitHub: Surjeet4146)

2

Hi, Surjeet. Have you tried coding these state-space solutions? They often come with annealing-type tuning parameters that can be really challenging to tune in practice if you really want draws according to an updated posterior each iteration.

Rather than re-running full MCMC, you can warm start with a mass matrix and step size and initial positions from the previous run, which should make things much much faster. To some extent, this is what things like sequential Monte Carlo particle filters do. The general problem is called “filtering” in the signal processing literature, hence the names “Kalman filter” and “particle filter”.