With the proliferation of mobile networks, we face strong diversification of services, demanding the current network to embed more flexibility. To satisfy this daring need, network slicing is embraced as a promising solution for resource utilization, in 5G and future networks. In network slicing, dynamic resource orchestration and network slice management are critical for resource efficiency. However, it is highly complicated such that the traditional approaches can not effectively perform resource orchestration due to the lack of accurate models and hidden problem structures. To address this challenge, we propose a constrained reinforcement learning based approach for network slicing. We formulate the resource allocation problem as a Constrained Markov Decision Process (CMDP) and solve it using constrained reinforcement learning algorithms. Specifically, we use the adaptive interior-point policy optimization and policy safety layer methods to deal with cumulative and instantaneous constraints. Our evaluations show that our method is effective in resource allocation with service demand guarantees and significantly outperforms baselines.