The continuous depletion of fossil fuels and the increasing demand for eco-friendly and sustainable energy sources have prompted researchers to look for alternative energy sources. The loss of thermal energy in heat engines (100 C–350 C), coal-based thermal plants (150 C–700 C), heated water pumping in the geothermal process (150 C–700 C), and burning of petrol in the automobiles (150 C–250 C) in form of untapped waste-heat can be directly and/or reversibly converted into usable electricity by means of charge carriers (electrons or holes) as moving fluids using thermoelectric (TE) technology, which works based on typical Seebeck effect. The enhancement in TE conversion efficiency has been a key challenge because of the coupled relation between thermal and electrical transport of charge carriers in a given material. In this review, we have deliberated the physical concepts governing the materials to device performance as well as key challenges for enhancing the TE performance. Moreover, the role of crystal structure in the form of chemical bonding, crystal symmetry, order–disorder and phase transition on charge carrier transport in the material has been explored. Further, this review has also emphasized some insights on various approaches employed recently to improve the TE performance, such as,(i) carrier engineering via band engineering, low dimensional effects, and energy filtering effects and (ii) phonon engineering via doping/alloying, nano-structuring, embedding secondary phases in the matrix and microstructural engineering. We have also briefed the importance of magnetic elements on thermoelectric properties of the selected materials and spin Seebeck effect. Furthermore, the design and fabrication of TE modules and their major challenges are also discussed. As, thermoelectric figure of merit, zT does not have any theoretical limitation, an ideal high performance thermoelectric device should consist of low-cost, eco-friendly, efficient, n-or p-type materials that operate at wide-temperature range and similar coefficients of thermal expansion, suitable contact materials, less electrical/thermal losses and constant source of thermal energy. Overall, this review provides the recent physical concepts adopted and fabrication procedures of TE materials and device so as to improve the fundamental understanding and to develop a promising TE device.