Wheelset　fault　detection　with　high　accuracy　is　challenging　due　to　poor　image　quality.　Specifically,　the　wheelset　images　are　collected　dynamically　outdoors　and　suffer　from　diffuse　reflection　and　environmental　interference.　Thus,　the　images　contain　light　stripe　adhesions　(light　flairs)　and　local　fractures　to　be　inpainted.　The　existing　inpainting　models　are　inapplicable　to　restore　grayscale　wheelset　images.　They　are　also　too　heavy　to　be　deployed　in　an　embedded　wheelset　monitoring　equipment.　In　this　paper,　we　propose　a　lightweight　high-precision　inpainting　model　that　consists　of　a　recurrent　similarity　network　with　the　ghost　convolution　(RSG-Net)　to　remove　light　flairs　and　repair　local　fractures.　RSG-Net　replaces　standard　Pconv　(partial　convolutional)　layers　with　soft-coding　ones　that　can　improve　the　feature　representational　ability.　To　reduce　the　influence　of　the　background　region　features　on　image　restoration,　an　asymmetrical　similarity　measure　is　designed　to　calculate　not　only　the　angle　difference　between　the　target　and　the　source　feature　vectors　but　also　the　activation　of　the　source　ones.　The　multi-scale　structural　similarity　(MS-SSIM)　loss　term　is　introduced　to　precisely　guide　the　structural　information　restoration,　such　as　the　stripe　edges.　Moreover,　the　ghost　convolution　is　introduced　in　RSG-Net　to　realize　the　model　compression　that　can　retain　the　core　features　of　wheelset　images　and　remove　the　redundant　features.　We　conduct　three　groups　of　experiments　that　demonstrate　the　accuracy　superiority　of　the　proposed　RSG-Net　over　the　baseline　methods,　and　the　number　of　parameters　is　reduced　by　about　50%.