Soiling　can　reduce　the　output　power　and　work　efficiency　of　photovoltaic　(PV)　modules,　causing　serious　economic　losses　to　PV　systems.　The　cleaning　schedules　can　be　optimized　to　save　economic　expenses　through　the　methods　capable　of　estimating　the　power　loss　of　PV　modules　resulting　from　soiling.　This　paper　proposes　a　deep　learning　framework　that　combines　visible　light　and　infrared　image　information　with　dual　branch　cross-modality　feature　fusion.　Initially,　the　MobileNetV2　is　applied　as　the　backbone　of　the　dual　branch　framework　to　enhance　the　training　efficiency　and　reduce　the　computational　complexity.　Subsequently,　a　cross-modality　differential　aware　fusion　module　based　on　the　channel　attention　mechanism　(CA-CMDAF)　is　introduced　to　improve　the　crossmodality　feature　fusion　capability　of　the　model.　Moreover,　a　multi-cascade　and　cross-modality　fusion　network　and　a　multi-scale　fusion　network　are　integrated　to　further　facilitate　the　effectiveness　of　feature　fusion　and　reduce　the　loss　of　visual　details　during　the　feature　extraction.　Lastly,　extensive　experiments　are　carried　out　on　the　multimodality　dataset.　The　comparison　results　demonstrate　the　superior　performance　of　the　proposed　dual　branch　network　framework　with　the　average　accuracy　of　88.27　%,　which　is　higher　than　that　of　the　single-modality　models　trained　on　either　visible　light　or　infrared　images　alone.