Robust　moving　object　pose　estimation　is　crucial　in　fine　manipulation　tasks,　such　as　surgical　instrument　tracking.　This　paper　presents　a　distributed-camera　system　with　robotic　adjustments　to　maintain　consistent　tracking　of　moving　objects,　thus　avoiding　tracking　failures.　An　integrated　framework　for　camera　adjustment　and　pose　estimation　is　developed　for　this　distributed-camera　system.　In　each　detection　cycle,　the　camera　exhibiting　the　largest　deviation　with　the　object　is　adjusted　by　a　visual　servoing　technique.　After　adjustment,　the　camera　extrinsics　are　re-calibrated　in　the　following　detection　cycles.　For　the　unadjusted　cameras,　an　online　extrinsic　optimization　method　based　on　multi-frame　detection　results　is　proposed　to　refine　the　camera　extrinsics.　Based　on　the　refined　camera　extrinsics　and　detection　results　from　multiple　cameras,　the　pose　of　moving　objects　relative　to　the　principal　camera　can　be　robustly　estimated.　We　test　the　performance　of　this　system　in　both　simulation　environments　and　real-world　scenarios.　The　results　indicate　that　our　system　achieves　higher　pose　estimation　accuracy　and　exhibits　strong　resistance　to　limited　field-of-view　(FoV)　compared　to　conventional　equivalent　fixed　multi-camera　systems.