The neural bases of action and abstract concept representations remain a topic of debate. While several lines of research provide evidence for grounding of action-related conceptual content into sensory-motor systems, results of traditional lesion-deficit studies have been somewhat inconsistent. Further, few studies have directly compared the neural substrates of action and relatively abstract verb comprehension post-stroke. Here, we investigated the impact of the disruption of two neural networks on comprehension of action and relatively abstract verbs in 48 unilateral left-hemisphere stroke patients using two methodologies: 1) lesion-deficit association and 2) resting-state functional connectivity (RSFC) analyses. Disruption of RSFC between the left inferior frontal gyrus and right hemisphere primary and secondary sensory-motor areas predicted greater relative impairment of action semantics. Voxel-based lesion-symptom mapping revealed that damage to frontal white matter, extending towards the inferior frontal gyrus, also predicted greater relative impairment of action semantics. On the other hand, damage to the left anterior middle temporal gyrus significantly impaired the more abstract category relative to action. These findings support the view that action and non-action/abstract semantic processing rely on partially dissociable brain networks, with action concepts relying more heavily on sensory-motor areas. The results also have wider implications for lesion-deficit association studies and show how the contralateral hemisphere can play a compensatory role following unilateral stroke.