Ultra-dense　mobile　networks　(UDMNs)　represent　a　promising　technology　for　improving　the　network　performance　and　providing　the　ubiquitous　network　accessibility　in　the　beyond　5　G　(B5G)　mobile　networks.　Heterogenous　densely　deployed　networks　can　dynamically　offer　high　spectrum　efficiency　and　enhance　frequency　reuse,　which　ultimately　improves　quality　of　service　(QoS)　and　the　user　experience.　However,　mass　inter-　or　intra-cell　interference　generated　from　overlap　between　small　cells　greatly　limits　network　performance,　especially　when　there　is　mobility　between　UEs　and　access　points　(APs).　Even　so,　when　network　density　increases,　the　complexity　of　conventional　allocation　methods　can　increase　also.　In　this　paper,　we　investigate　a　power　control　of　downlink　(DL)　connection　in　the　UNMNs　with　different　types　of　APs.　We　propose　a　reinforcement　learning　(RL)　power　allocation　algorithm　based　on　graph　signal　processing　(GSP)　for　ultra-dense　mobile　networks.　Firstly,　we　construct　a　realistic　system　model　under　ultra-dense　mobile　networking,　which　includes　the　system　channel　mode　and　instantaneous　rate.　Then　we　employ　a　GSP　tool　to　analyze　network　interference,　the　interference　analysis　results　for　the　entire　network　are　obtained　to　determine　optimal　RL　power　allocation.　Finally,　simulation　results　indicate　that　the　proposed　RL　power　control　algorithm　outperforms　baseline　algorithms　when　applied　to　a　ultra-dense　mobile　networks.