It has been postulated that rare coding variants (RVs; MAF < 0.01) contribute to the “missing” heritability of complex traits. We developed a framework, the Rare variant heritability (RARity) estimator, to assess RV heritability (h²_RV) without assuming a particular genetic architecture. We applied RARity to 31 complex traits in the UK Biobank (n = 167,348) and showed that gene-level RV aggregation suffers from 79% (95% CI: 68-93%) loss of h²_RV. Using unaggregated variants, 27 traits had h²_RV > 5%, with height having the highest h²_RV at 21.9% (95% CI: 19.0-24.8%). The total heritability, including common and rare variants, recovered pedigree-based estimates for 11 traits. RARity can estimate gene-level h²_RV, enabling the assessment of gene-level characteristics and revealing 11, previously unreported, gene-phenotype relationships. Finally, we demonstrated that in silico pathogenicity prediction (variant-level) and gene-level annotations do not generally enrich for RVs that over-contribute to complex trait variance, and thus, innovative methods are needed to predict RV functionality.