# How to implement learning rate decay?

**URL:** <https://discourse.pymc.io/t/how-to-implement-learning-rate-decay/12594>\
**Category:** v5\
**Tags:** modeling\
**Created:** [July 26, 2023, 3:59am UTC](https://discourse.pymc.io/t/how-to-implement-learning-rate-decay/12594 "2023-07-26T03:59:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dushyant](https://avatars.discourse-cdn.com/v4/letter/d/e480ec/32.png) [@dushyant](https://discourse.pymc.io/u/dushyant)\
**Post date:** [July 26, 2023, 3:59am UTC](https://discourse.pymc.io/t/how-to-implement-learning-rate-decay/12594/1 "2023-07-26T03:59:44Z")

</div>

I am stuck in a scenario where the learning rate of adam for ADVI needs to decrease with the number of epochs. Towards convergence, the optimizer bounces around a lot resulting in very different results. I want to decrease the learning rate as the optimizer converges so that the bouncing behavior reduces and converges to a stable optimum. What would be the best way to implement this? Should I directly change the code in the library and then import it, or is there a better way?

---

<div class="post-metadata">

**Author:** ![dushyant](https://avatars.discourse-cdn.com/v4/letter/d/e480ec/32.png) [@dushyant](https://discourse.pymc.io/u/dushyant)\
**Post date:** [July 27, 2023, 11:42pm UTC](https://discourse.pymc.io/t/how-to-implement-learning-rate-decay/12594/2 "2023-07-27T23:42:43Z")

</div>

Modified the PyMC’s Adam code and worked perfectly fine.