Summary:
doesn't affect speed, but may save a bit more memory

Pull Request resolved: https://github.com/fairinternal/detectron2/pull/482

Test Plan:
f233140419

f233150179

RetinaNet (on fair cluster): 0.14s/step, memory 3427M, mAP is good.

Reviewed By: theschnitz

Differential Revision: D25003984

Pulled By: ppwwyyxx

fbshipit-source-id: c251b6da1225e20b338775aae3e5882163a4015b