Calculation of Average Loss in variational/inference.py


#1

Hi,

I tried to follow exactly how the output “Average Loss = XXX” is calculated because it didn’t quite match what I calculated manually. Looking into _iterate_with_loss (https://github.com/pymc-devs/pymc3/blob/master/pymc3/variational/inference.py) raised two questions in my head that I would like to clarify.

Why is the progress description updated twice in one update cycle github?

avg_loss = _infmean(scores[max(0, i - 1000):i + 1])
progress.set_description('Average Loss = {:,.5g}'.format(avg_loss))
avg_loss = scores[max(0, i - 1000):i + 1].mean()
progress.set_description(
    'Average Loss = {:,.5g}'.format(avg_loss))

Why is the avg_loss calculation different in line 201? In most of the cases this will calculate the avg loss as mean over the whole loss history, wheares in the other places it is calculated as mean over the last 1000 losses:

avg_loss = _infmean(scores[min(0, i - 1000):i + 1]) # compared to
avg_loss = _infmean(scores[max(0, i - 1000):i + 1])