Wood　surface　broken　defects　seriously　damage　the　structure　of　wooden　products,　these　defects　have　to　be　detected　and　eliminated.　However,　current　defect　detection　methods　based　on　machine　vision　have　difficulty　distinguishing　the　interference,　similar　to　the　broken　defects,　such　as　stains　and　mineral　lines,　and　can　result　in　frequent　false　detections.　To　address　this　issue,　a　multi-source　data　fusion　network　based　on　U-Net　is　proposed　for　wood　broken　defect　detection,　combining　image　and　depth　data,　to　suppress　the　interference　and　achieve　complete　segmentation　of　the　defects.　To　efficiently　extract　various　semantic　information　of　defects,　an　improved　ResNet34　is　designed　to,　respectively,　generate　multi-level　features　of　the　image　and　depth　data,　in　which　the　depthwise　separable　convolution　(DSC)　and　dilated　convolution　(DC)　are　introduced　to　decrease　the　computational　expense　and　feature　redundancy.　To　take　full　advantages　of　two　types　of　data,　an　adaptive　interacting　fusion　module　(AIF)　is　designed　to　adaptively　integrate　them,　thereby　generating　accurate　feature　representation　of　the　broken　defects.　The　experiments　demonstrate　that　the　multi-source　data　fusion　network　can　effectively　improve　the　detection　accuracy　of　wood　broken　defects　and　reduce　the　false　detections　of　interference,　such　as　stains　and　mineral　lines.