Multi-atlas　segmentation　(MAS)　is　a　promising　framework　for　medical　image　segmentation.　Generally,　MAS　methods　register　multiple　atlases,　i.e.,　medical　images　with　corresponding　labels,　to　a　target　image;　and　the　transformed　atlas　labels　can　be　combined　to　generate　target　segmentation　via　label　fusion　schemes.　Many　conventional　MAS　methods　employed　the　atlases　from　the　same　modality　as　the　target　image.　However,　the　number　of　atlases　with　the　same　modality　may　be　limited　or　even　missing　in　many　clinical　applications.　Besides,　conventional　MAS　methods　suffer　from　the　computational　burden　of　registration　or　label　fusion　procedures.　In　this　work,　we　design　a　novel　cross-modality　MAS　framework,　which　uses　available　atlases　from　a　certain　modality　to　segment　a　target　image　from　another　modality.　To　boost　the　computational　efficiency　of　the　framework,　both　the　image　registration　and　label　fusion　are　achieved　by　well-designed　deep　neural　networks.　For　the　atlas-to-target　image　registration,　we　propose　a　bi-directional　registration　network　(BiRegNet),　which　can　efficiently　align　images　from　different　modalities.　For　the　label　fusion,　we　design　a　similarity　estimation　network　(SimNet),　which　estimates　the　fusion　weight　of　each　atlas　by　measuring　its　similarity　to　the　target　image.　SimNet　can　learn　multi-scale　information　for　similarity　estimation　to　improve　the　performance　of　label　fusion.　The　proposed　framework　was　evaluated　by　the　left　ventricle　and　liver　segmentation　tasks　on　the　MM-WHS　and　CHAOS　datasets,　respectively.　Results　have　shown　that　the　framework　is　effective　for　cross-modality　MAS　in　both　registration　and　label　fusion　https://github.com/NanYoMy/cmmas.