An important step in the processing of visual patterns is the segmentation of the retinal image. Neuronal responses evoked by the contours of individual objects need to be identified and associated for further joint processing. These grouping operations are based on a number of Gestalt criteria. Here we report that connections in the visual cortex of the cat exhibit a highly significant anisotropy, preferentially linking neurons activated by contours that have similar orientation and are aligned colinearly. These anatomical data suggest a close relation between the perceptual grouping criterion of colinearity and the topology of tangential intracortical connections. We propose that tangential intracortical connections support perceptual grouping by modulating the saliency of distributed cortical responses in a context‐dependent way. The present data are compatible with the hypothesis that the criteria for this grouping operation are determined by the architecture of the tangential connections.