Building　extraction　from　remote　sensing　imagery　is　vital　for　various　human　activities.　But　it　is　challenging　due　to　diverse　building　appearances　and　complex　backgrounds.　Research　shows　the　importance　of　both　global　context　and　spatial　details　for　accurate　building　extraction.　Therefore,　methods　integrating　convolutional　neural　networks　(CNNs)　and　visual　transformers　(ViTs)　are　popular　nowadays.　However,　current　methods　combining　these　two　methods　inadequately　merge　their　features　and　only　perform　decoding　once,　leading　to　issues　like　unclear　boundaries,　internal　voids,　and　susceptibility　to　non-building　elements　in　complex　scenarios　with　low　inter-class　and　high　intra-class　variability.　To　address　these　issues,　this　paper　introduces　a　novel　extraction　method　called　TDFNet.　We　first　replace　ViT　with　V-Mamba,　which　has　linear　complexity,　and　combine　it　with　CNN　for　feature　extraction.　A　bidirectional　fusion　module　(BFM)　is　then　designed　to　comprehensively　integrate　spatial　details　and　global　information,　thereby　enabling　accurate　identification　of　boundaries　between　adjacent　buildings,　and　maintaining　the　structural　integrity　of　buildings　to　avoid　internal　holes.　During　the　decoding　process,　we　propose　an　Encoder-Decoder　Fusion　Module　(EDFM)　to　initially　merge　features　from　different　stages　of　the　encoder　and　decoder,　thereby　diminishing　the　model＇s　susceptibility　to　non-building　elements　with　features　similar　to　those　of　buildings,　and　consequently　reducing　the　incidence　of　erroneous　extractions.　Subsequently,　a　twice　decoding　strategy　is　implemented　to　enhance　the　learning　of　multi-scale　features　significantly,　thereby　mitigating　the　impact　of　tree　occlusions　and　shadows.　Our　method　yields　the　state-of-the-art　(SOTA)　performance　on　three　public　building　datasets.