Folklore suggests that policy gradient can be more robust to misspecification than its relative, approximate policy iteration. This paper studies the case of state-aggregated representations, in which the state space is partitioned and either the policy or value function approximation is held constant over partitions. This paper shows a policy gradient method converges to a policy whose regret per period is bounded by ϵ, the largest difference between two elements of the state-action value function belonging to a common partition. With the same representation, both approximate policy iteration and approximate value iteration can produce policies whose per-period regret scales as , where γ is a discount factor. Faced with inherent approximation error, methods that locally optimize the true decision objective can be far more robust.

Supplemental Material: Data are available at https://doi.org/10.1287/mnsc.2023.4788.