Panoramic video, features a high frame rate and high resolution, can provide viewers an immersive visual experience. However, it necessitates more overhead bits during video transmission, leading to a heavy burden on storage and bandwidth. Most of the existing panoramic video coding optimizations did not take into account the effect of pixel position on the Lagrange multiplier in the rate-distortion optimization (RDO) process. Meanwhile, those methods did not take into account the effect of the temporal impact factor on the RDO process. In this paper, we proposed a latitude-guided temporally dependent RDO algorithm for panoramic video. Firstly, the quantization parameter (QP) of the coding unit (CU) is determined by the latitude corresponding to the pixel. Secondly, the Lagrange multiplier of the corresponding coding tree unit (CTU) is calculated by the QP of the CTU. Finally, the Lagrange multiplier is fine-tuned to improve the RDO process by using the temporal dependency among CTUs. Experimental results show that, on the platform of HEVC test model (HM-16.14), the proposed algorithm obtains bitrate savings of 4.15%, 4.14% and 4.05% on average under the quality assessment criteria of Spherical PSNR (S-PSNR), Weighted to Spherically Uniform PSNR (WS-PSNR) and Craster Parabolic Projection PSNR (CPP-PSNR) for the equirectangular projection (ERP).