Hi Timothé,
I don’t know the inverse-Gamme very well, so, in general, I’d say that if you think it’s appropriate through your domain knowledge, then go for it!
My advice would just be to test it against an exponential prior, and see if inference changes. Coupled with prior pred checks and domain knowledge, this should give you a good idea of what an appropriate prior is in this case.
The maximum entropy principle could also help you decide. To quote McElreath (Statistical Rethinking 2nd ed., p.326):
“The exponential distribution has maximum entropy among all non-negative continuous distributions with the same average displacement. […] The gamma distribution has maximum entropy among all distributions with the same mean and same average logarithm.”
But I wouldn’t sweat on it, as Gamma and Exponential are related (the former is a sum of exp RVs).