i am new to PYMC and am working on an Auto regressive model. i would like to add l1 or l2 regularization to the model but couldn’t find an example or tutorial for it.
could someone point me to some relevant material or a solution maybe.
thanx
Hi Radheem,
I don’t have a definitive answer to your question – and I’m actually interested in the responses that more experienced people than me will give – but from what I understood I think you won’t have a one-to-one equivalence in the Bayesian framework.
I mean that you get that for free in Bayes: just putting sensible priors is a form of regularization, and, at a higher level, hierarchical models are very useful to regularize parameters and guard against both over and underfitting.
So I think it’s not a binary situation in the Bayesian framework – either you regularize or you don’t. It’s more like a knob that you adjust – you penalize parameters more or less, depending on your priors and use case.
Hope this helps and that I didn’t talk too much nonsense 
Hi @AlexAndorra
I get what you are saying but I am predicting a set of values for a time series problem. Let’s say I am predicting X for some day D. I have the actual value of X, let’s call it AX for day D and now I want to compare and reduce the difference between AX and X. So I was hoping to use L1 or L2 regularisation for that. Hope this makes sense and is actually a problem and not just some misconception.
Mmmh yeah ok I think it relates to loss functions in general, as L1, L2 or OLS are just different loss functions. I remember a discussion about that in @aloctavodia’s book, so I’m pinging him – I think he’ll have good advice on that 
For what it’s worth, L2 regularization is analogous to normal priors for residuals and L1 is analogous to Laplace prior (but really you should use (Finnish) horse-shoe prior).
See for example https://betanalpha.github.io/assets/case_studies/bayes_sparse_regression.html
What you should do, is to model the data generating process and set-up suitable priors for it.
3 Likes