In　the　algorithm　of　self-organizing　background　subtraction,　the　real-time　performance　is　poor　and　the　background　model　is　easy　to　offset　under　complicated　environment　due　to　non-periodic　changes.　Aiming　at　these　problems,　a　hierarchical　algorithm　of　self-organizing　background　subtraction　with　memory　storage　is　proposed.　Firstly,　a　border-shared　background　model　is　established　to　reduce　the　time　and　space　complexity.　And　a　cache　strategy　is　introduced　on　the　basis　of　the　original　matrix　model　to　store　the　past　and　the　current　background　data,　respectively.　Then,　during　the　object　detection,　a　decision　mechanism　for　different　granularity　levels　is　designed　to　determine　whether　a　pixel　is　an　object　or　not.　Experimental　results　show　that　the　proposed　algorithm　can　overcome　the　shortcomings　of　the　original　one　and　achieve　higher　detection　accuracy　and　better　real-time　performance.　©　2016,　Science　Press.　All　right　reserved.