Multimodal　representation　learning,　which　aims　to　narrow　the　heterogeneity　gap　among　different　modalities,　plays　an　indispensable　role　in　the　utilization　of　ubiquitous　multimodal　data.　Due　to　the　powerful　representation　ability　with　multiple　levels　of　abstraction,　deep　learning-based　multimodal　representation　learning　has　attracted　much　attention　in　recent　years.　In　this　paper,　we　provided　a　comprehensive　survey　on　deep　multimodal　representation　learning　which　has　never　been　concentrated　entirely.　To　facilitate　the　discussion　on　how　the　heterogeneity　gap　is　narrowed,　according　to　the　underlying　structures　in　which　different　modalities　are　integrated,　we　category　deep　multimodal　representation　learning　methods　into　three　frameworks:　joint　representation,　coordinated　representation,　and　encoder-decoder.　Additionally,　we　review　some　typical　models　in　this　area　ranging　from　conventional　models　to　newly　developed　technologies.　This　paper　highlights　on　the　key　issues　of　newly　developed　technologies,　such　as　encoder-decoder　model,　generative　adversarial　networks,　and　attention　mechanism　in　a　multimodal　representation　learning　perspective,　which,　to　the　best　of　our　knowledge,　have　never　been　reviewed　previously,　even　though　they　have　become　the　major　focuses　of　much　contemporary　research.　For　each　framework　or　model,　we　discuss　its　basic　structure,　learning　objective,　application　scenes,　key　issues,　advantages,　and　disadvantages,　such　that　both　novel　and　experienced　researchers　can　benefit　from　this　survey.　Finally,　we　suggest　some　important　directions　for　future　work.　©　2013　IEEE.