Crowd　image　is　arguably　one　of　the　most　laborious　data　to　annotate.　In　this　paper,　we　aim　to　reduce　the　massive　demand　for　densely　labeled　crowd　data,　and　propose　a　novel　weakly-supervised　setting,　in　which　we　leverage　the　binary　ranking　of　two　images　with　high-contrast　crowd　counts　as　training　guidance.　To　enable　training　under　this　new　setting,　we　convert　the　crowd　count　regression　problem　to　a　ranking　potential　prediction　problem.　In　particular,　we　tailor　a　Siamese　Ranking　Network　that　predicts　the　potential　scores　of　two　images　indicating　the　ordering　of　the　counts.　Hence,　the　ultimate　goal　is　to　assign　appropriate　potentials　for　all　the　crowd　images　to　ensure　their　orderings　obey　the　ranking　labels.　On　the　other　hand,　potentials　reveal　the　relative　crowd　sizes　but　cannot　yield　an　exact　crowd　count.　We　resolve　this　problem　by　introducing　＂anchors＂　during　the　inference　stage.　Concretely,　anchors　are　a　few　images　with　count　labels　used　for　referencing　the　corresponding　counts　from　potential　scores　by　a　simple　linear　mapping　function.　We　conduct　extensive　experiments　to　study　various　combinations　of　supervision,　and　we　show　that　our　method　outperforms　existing　weakly-supervised　methods　by　a　large　margin　without　additional　labeling　effort.　The　code　is　available　at　https://github.com/pandaszzzzz/CCRanking.