Recent legislation mandates that the US Food and Drug Administration issue guidance regarding when real‐world evidence (RWE) could be used to support regulatory decision making. Although RWE could come from randomized or nonrandomized designs, there are significant concerns about the validity of RWE assessing medication effectiveness based on nonrandomized designs. We propose an initiative using healthcare claims data to assess the ability of nonrandomized RWE to provide results that are comparable with those from randomized controlled trials (RCTs). We selected 40 RCTs, and we estimate that approximately 30 attempted replications will be completed after feasibility analyses. We designed an implementation process to ensure that each attempted replication is consistent, transparent, and reproducible. This initiative is the first to systematically evaluate the ability of nonrandomized RWE to replicate multiple RCTs using a structured process. Results from this study should provide insight on the strengths and limitations of using nonrandomized RWE from claims for regulatory decision making.