Batch　Normalization　(BN)　techniques　have　been　proposed　to　reduce　the　so-called　Internal　Covariate　Shift　(ICS)　by　attempting　to　keep　the　distributions　of　layer　outputs　unchanged.　Experiments　have　shown　their　effectiveness　on　training　deep　neural　networks.　However,　since　only　the　first　two　moments　are　controlled　in　these　BN　techniques,　it　seems　that　a　weak　constraint　is　imposed　on　layer　distributions　and　furthermore　whether　such　constraint　can　reduce　ICS　is　unknown.　Thus　this　paper　proposes　a　measure　for　ICS　by　using　the　Earth　Mover　(EM)　distance　and　then　derives　the　upper　and　lower　bounds　for　the　measure　to　provide　a　theoretical　analysis　of　BN.　The　upper　bound　has　shown　that　BN　techniques　can　control　ICS　only　for　the　outputs　with　low　dimensions　and　small　noise　whereas　their　control　is　not　effective　in　other　cases.　This　paper　also　proves　that　such　control　is　just　a　bounding　of　ICS　rather　than　a　reduction　of　ICS.　Meanwhile,　the　analysis　shows　that　the　high-order　moments　and　noise,　which　BN　cannot　control,　have　great　impact　on　the　lower　bound.　Based　on　such　analysis,　this　paper　furthermore　proposes　an　algorithm　that　unitizes　the　outputs　with　an　adjustable　parameter　to　further　bound　ICS　in　order　to　cope　with　the　problems　of　BN.　The　upper　bound　for　the　proposed　unitization　is　noise-free　and　only　dominated　by　the　parameter.　Thus,　the　parameter　can　be　trained　to　tune　the　bound　and　further　to　control　ICS.　Besides,　the　unitization　is　embedded　into　the　framework　of　BN　to　reduce　the　information　loss.　The　experiments　show　that　this　proposed　algorithm　outperforms　existing　BN　techniques　on　CIFAR-10,　CIFAR-100　and　ImageNet　datasets.　©　2020　IEEE.