Rate　distortion　optimization　(RDO)　is　an　important　approach　for　ensuring　the　coding　efficiency　of　the　encoder　by　selecting　the　optimal　combination　of　multiple　candidate　coding　parameters　in　video　coding.　Discrete　sine　transform　(DST)　has　high　accuracy　but　causes　high　complexity　in　RDO,　whereas　Hadamard　transform　is　simple　but　unsuitable　for　estimating　the　transform　coefficients　of　DST　in　RDO.　In　this　study,　we　initially　proposed　a　new　simplified　DST　transformation　matrix　to　estimate　the　rate　distortion　(RD)　cost　accurately　for　intra　prediction　and　then　analyzed　the　decorrelated　performance　and　complexity.　Moreover,　we　designed　a　high-throughput　hardware　architecture　based　on　the　FPGA　platform.　Experimental　results　show　that　in　comparison　with　the　high　efficiency　video　coding　reference　software　HM16.7,　the　proposed　simplified　DST　matrix-based　RD　cost　estimation　model　improves　the　RD　performance.　Moreover,　the　proposed　architecture　achieves　higher　throughput　with　pipeline　structure　compared　with　the　existing　hardware　structures.　The　implementation　in　Altera＇s　Arria　10　FPGA　can　operate　at　275　MHz　and　supports　real-time　processing　of　4096×2160　ultra-high-definition　at　a　minimum　of　75　fps.　©　2019　The　authors　and　IOS　Press.　All　rights　reserved.