The Mars Helicopter (MH) will be flying on the NASA Mars 2020 rover mission scheduled to launch in July of 2020. Research is being performed at the Jet Propulsion Laboratory (JPL) and NASA Ames Research Center to extend the current capabilities and develop the Mars Science Helicopter (MSH) as the next possible step for Martian rotorcraft. The low atmospheric density and the relatively small-scale rotors result in very low chord-based Reynolds number flows over the rotor airfoils. The low Reynolds number regime results in rapid performance degradation for conventional airfoils due to laminar separation without reattachment. Unconventional airfoil shapes with sharp leading edges are explored and optimized for aerodynamic performance at representative Reynolds-Mach combinations for a concept rotor. Sharp leading edges initiate immediate flow separation, and the occurrence of large-scale vortex shedding is found to contribute to the relative performance increase of the optimized airfoils, compared to conventional airfoil shapes. The oscillations are shown to occur independent from laminar-turbulent transition and therefore result in sustainable performance at lower Reynolds numbers. Comparisons are presented to conventional airfoil shapes and peak lift-to-drag ratio increases between 17% and 41% are observed for similar section lift.