Is your grad operation performed on matrix? I am a bit confuse - why not take gradient of each input directly?