We present higher-spin algebras containing a Poincaré subalgebra and with the same set of generators as the Lie algebras that are relevant to Vasiliev’s equations in any space-time dimension D≥ 3. Given these properties, they can be considered either as candidate rigid symmetries for higher-spin gauge theories in Minkowski space or as Carrollian conformal higher-spin symmetries in one less dimension. We build these Lie algebras as quotients of the universal enveloping algebra of and we show how to recover them as Inönü-Wigner contractions of the rigid symmetries of higher-spin gauge theories in Anti de Sitter space or, equivalently, of relativistic conformal higher-spin symmetries. We use the same techniques to also define higher-spin algebras with the same set of generators and containing a Galilean conformal subalgebra, to be interpreted as non-relativistic limits of the conformal symmetries of a free scalar field. We begin by showing that the known flat-space higher-spin algebras in three dimensions can be obtained as quotients of the universal enveloping algebra of and then we extend the analysis along the same lines to a generic number of space-time dimensions. We also discuss the peculiarities that emerge for D= 5.