This paper describes a vision chip architecture for high-speed vision systems that we propose. The chip has general-purpose processing elements (PEs) in massively parallel architecture, with each PE directly connected to photo-detectors. Control programs allow various visual processing applications and algorithms to be implemented. A sampling rate of 1 ms is enough to realize high-speed visual feedback for robot control. To integrate as many PEs as possible on a single chip a compact design is required, so we aim to create a very simple architecture. The sample design has been implemented into an FPGA chip; a full custom chip has also been designed and has been submitted for fabrication.