The verification and validation (V&V) of autonomous systems is a complex and difficult task, especially when artificial intelligence is used to achieve autonomy. However, without proper V&V, sufficient evidence to argue safety is not attainable. We propose in this work the use of a Safety Supervisor (SSV) to circumvent this issue. However, the design of an adequate SSV is a challenge in itself. To assist in this task, we present a conceptual framework and a corresponding metamodel, which are motivated and justified by existing work in the field. The conceptual framework supports the alignment of future research in the field of runtime safety monitoring. Our vision is for the different parts of the framework to be filled with exchangeable solutions so that a concrete SSV can be derived systematically and efficiently, and that new solutions can be embedded in it and get evaluated against existing approaches. To exemplify our vision, we present an SSV that is based on the ISO 22839 standard for forward collision mitigation.