Identifying　weak　fault　features　caused　by　an　incipient　tooth　crack　in　a　planetary　gearbox　is　difficult.　To　characterize　the　fault　features　of　incipient　tooth　cracks　in　a　planetary　gear　train,　this　paper　established　a　transmission-structure-coupled　nonlinear　dynamic　model　that　includes　the　effect　of　crack　damage　with　lumped　parameter　method.　Based　on　potential　energy　method,　a　mathematical　model　of　time-varying　mesh　stiffness　and　transmission　error　was　established.　The　effect　of　crack　damage　was　incorporated　into　the　nonlinear　dynamic　transmission　model　through　stiffness　excitation　and　displacement　excitation　functions.　Furthermore,　the　vibration　response　of　the　planetary　transmission　system　was　solved,　indicating　that　the　difference　in　transmission　error　between　internal　and　external　transmission　branches　is　an　important　cause　of　non-uniform　load　distribution.　Moreover,　a　finite　element　model　of　the　gearbox　was　built　in　the　ANSYS　Workbench　environment.　After　applying　the　reaction　forces　of　the　sun　gear,　the　carrier　and　the　ring　gear　on　the　bearing　housings　of　the　gearbox,　and　including　the　modulation　effect　of　the　carrier　rotation　with　a　window　function,　frequency　analysis　can　be　conducted　on　the　simulated　vibration　signals　to　identify　the　damage　mode　of　the　tooth　crack.　Finally,　a　test　rig　of　the　planetary　gearbox　was　constructed,　based　on　which　a　set　of　vibration　tests　were　conducted　to　detect　the　tooth　crack.　The　consistencies　of　the　simulated　and　experimental　signals　prove　that　the　proposed　transmission-structure-coupled　dynamic　model　can　accurately　reveal　the　fault　mechanism　of　the　planetary　gearbox　with　an　incipient　tooth　crack.　The　fault　features　of　this　planetary　gearbox　demonstrate　a　set　of　modulated　sidebands　around　the　mesh　frequency.　Additionally,　the　intervals　of　the　modulated　sidebands　are　the　fractional　frequencies　doubling　of　the　fault　frequency　and　carrier　rotation　frequency.　©　2019,　Editorial　Board　of　Journal　of　Tianjin　University(Science　and　Technology).　All　right　reserved.