L1 and L2 pegularization for an Autoregressive model

i am new to PYMC and am working on an Auto regressive model. i would like to add l1 or l2 regularization to the model but couldn’t find an example or tutorial for it.
could someone point me to some relevant material or a solution maybe.

Hi Radheem,
I don’t have a definitive answer to your question – and I’m actually interested in the responses that more experienced people than me will give – but from what I understood I think you won’t have a one-to-one equivalence in the Bayesian framework.

I mean that you get that for free in Bayes: just putting sensible priors is a form of regularization, and, at a higher level, hierarchical models are very useful to regularize parameters and guard against both over and underfitting.
So I think it’s not a binary situation in the Bayesian framework – either you regularize or you don’t. It’s more like a knob that you adjust – you penalize parameters more or less, depending on your priors and use case.

Hope this helps and that I didn’t talk too much nonsense :stuck_out_tongue_winking_eye:

Hi @AlexAndorra
I get what you are saying but I am predicting a set of values for a time series problem. Let’s say I am predicting X for some day D. I have the actual value of X, let’s call it AX for day D and now I want to compare and reduce the difference between AX and X. So I was hoping to use L1 or L2 regularisation for that. Hope this makes sense and is actually a problem and not just some misconception.:hugs:

Mmmh yeah ok I think it relates to loss functions in general, as L1, L2 or OLS are just different loss functions. I remember a discussion about that in @aloctavodia’s book, so I’m pinging him – I think he’ll have good advice on that :wink:

For what it’s worth, L2 regularization is analogous to normal priors for residuals and L1 is analogous to Laplace prior (but really you should use (Finnish) horse-shoe prior).

See for example https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html

What you should do, is to model the data generating process and set-up suitable priors for it.