We　present　a　context　aware　object　detection　method　based　on　a　retrieve-and-transform　scene　layout　model.　Given　an　input　image,　our　approach　first　retrieves　a　coarse　scene　layout　from　a　codebook　of　typical　layout　templates.　In　order　to　handle　large　layout　variations,　we　use　a　variant　of　the　spatial　transformer　network　to　transform　and　refine　the　retrieved　layout,　resulting　in　a　set　of　interpretable　and　semantically　meaningful　feature　maps　of　object　locations　and　scales.　The　above　steps　are　implemented　as　a　Layout　Transfer　Network　which　we　integrate　into　Faster　RCNN　to　allow　for　joint　reasoning　of　object　detection　and　scene　layout　estimation.　Extensive　experiments　on　three　public　datasets　verified　that　our　approach　provides　consistent　performance　improvements　to　the　state-of-the-art　object　detection　baselines　on　a　variety　of　challenging　tasks　in　the　traffic　surveillance　and　the　autonomous　driving　domains.