Average Loss in optimization output

So I’ve run into a situation where the loss first decreases with number of samples then increases. Should I assume that the model deteriorated based on this? (something in me says “no” but I can’t justify the intuition)