Emerging　multimedia　technologies　significantly　enhance　the　naturalness　and　immersion　of　human　computer　interaction.　Currently,　research　on　kinesthetic　information　has　gained　increasing　attentions　of　multimedia　community.　However,　effective　interaction　between　kinesthetic　and　other　multimedia　signals　remains　a　challenging　task.　In　this　paper,　we　propose　a　visual-kinesthetic　interaction　in　Virtual　Reality　(VR)　and　real-world　control　tasks.　First,　we　model　the　correlation　between　user’s　visual　attention　and　kinesthetic　positions　under　different　tasks.　Second,　we　utilize　an　attention-based　Long　Short-Term　Memory　network　to　predict　the　kinesthetic　positions.　Third,　we　build　a　VR　system　with　robotic　car　control,　which　validates　our　model　in　VR　interaction　and　control　tasks.　With　a　high　task　achievement　rate,　we　envision　the　implementation　of　kinesthetic　information　in　a　more　natural　interaction　system.　The　VR　interaction　system　based　on　the　proposed　model　can　also　provide　guidance　for　the　design　of　immersive　robot　teleoperation　systems.　©　2023,　The　Author(s),　under　exclusive　licence　to　Springer-Verlag　London　Ltd.,　part　of　Springer　Nature.