In　this　paper,　we　try　to　address　the　joint　optimization　problem　of　the　extreme　learning　machines　corresponding　to　different　features.　The　method　is　based　on　the　L　2,1　norm　penalty,　which　encourages　joint　sparse　coding.　By　adopting　such　a　technology,　the　intrinsic　relation　between　different　features　can　be　sufficiently　preserved.　To　tackle　the　problem　that　the　labeled　samples　is　rare,　we　introduce　the　semi-supervised　regularization　term　and　seamlessly　incorporate　them　into　the　particle　filter　framework　to　realize　visual　tracking.　In　addition,　an　online　updating　strategy　is　introduced　which　also　exploits　the　large　amount　of　unlabeled　samples　that　are　collected　during　the　tracking　period.　Finally,　the　proposed　tracking　algorithm　is　compared　to　other　state-of-the-arts　on　some　challenging　video　sequences　and　shows　promising　results.　©　2014　Springer　Science+Business　Media　New　York.