The unstable nonlinear Schrödinger models illustrate the time evolution of disturbances in marginally stable or unstable media. In this article, an improved auxiliary equation method is proposed for conformable fractional-order nonlinear evolution equations to obtain the novel wave solutions of space–time fractional unstable and modified unstable nonlinear Schrödinger equations. Several types of novel wave solutions are constructed in forms of some special Jacobi elliptic functions, rational, exponential, trigonometric and hyperbolic functions in which various are unique and having key utilization in quantum physics. With the aid of appropriate values to parameters, different shapes of bright, dark, kink type soliton, multi-peak solitons, periodic solitary waves etc. are depicted. These distinct physical structures are helpful in the understanding the complex physical interpretation of unstable dynamical models. Furthermore, this article gives an idea, how can reduce the conformable fractional order unstable nonlinear Schrödinger equations into an ODE of one variable to obtain the exact solutions. The numerous obtained waves and other solutions reveal the usefulness of the proposed technique which can be utilized to other nonlinear fractional models arising in quantum physics and other fields of applied sciences.