Summary: the gradients do not backprop to inputs - causing a deadlock.

Reviewed By: rbgirshick

Differential Revision: D20531868

fbshipit-source-id: fac23221decbe9e5740ac699b96ac08fabe4f832