This　paper　proposes　an　improved　null-space　behavior　control　algorithm　based　on　rotation　matrix　constraints　for　nonlinear　systems.　We　take　the　kinematic　model　of　unmanned　aerial　vehicles　(UAVs)　as　an　example.　First,　the　UAV　mathematical　model　is　incorporated　into　the　mission　design　of　null-space　behavior　control　(NSBC)　to　ensure　that　the　mission　instructions　satisfy　the　system＇s　motion　constraints,　overcoming　the　limitations　of　the　classical　NSBC　method,　which　relies　on　the　single　integrator　model.　Second,　a　formation　controller　was　designed　for　each　agent,　which　utilizes　a　self-updating　estimator　that　tracks　the　expected　speed　after　merging　under　disturbance　conditions.　Finally,　numerical　simulations　indicate　that　the　proposed　algorithm　can　form　a　rigid　formation.　©　Published　under　licence　by　IOP　Publishing　Ltd.