Thank you for your response: you are spot on! The ordering of the list given to the vars argument determines the order that grad_logp returns the gradients.
grad_logp = simple_gaussian.compile_dlogp(vars=[mu, log_tau])
grad_logp({"mu": current_mu, "log_tau": current_log_tau})