In this paper, we propose the implicit stabilized dual-horizon peridynamics-based strain gradient damage model (GDH-PD) to describe the cross-scale fracture behavior of materials. To this end, firstly, the strain energy density function of GDH-PD is reformulated by considering the energy compensation to eliminate zero-energy modes of the traditional higher-order peridynamics. And then, the constitutive force state of GDH-PD is derived and explicitly expressed with the help of the proposed special dimension reduction of the nonlocal higher-order tensors. To solve the steady-state crack propagation problems, the implicit GDH-PD is developed by deriving the lower- and higher-order micro-modulus double state, such that the linearization of the equilibrium equation of GDH-PD is established. At last, the bond length-dependent energy-based failure criterion is used to characterize the cross-scale fracture in the form of bond breakage. The effectiveness of GDH-PD to characterize microstructure size effects and macrostructure strain gradient effects are investigated by numerical simulations. The numerical results are in good agreement with the analytical solutions or the available experimental results. We believe that the proposed GDH-PD may pave the way to an increased application of peridynamics to be used in the cross-scale fracture predictions for the advanced material.