Due to the target of carbon neutrality and the current energy crisis in the world, green, flexible and low-cost distributed photovoltaic power generation is a promising trend. With battery energy storage to cushion the fluctuating and intermittent photovoltaic (PV) output, the photovoltaic battery (PVB) system has been getting increasing attention. This study is conducted to comprehensively review the PVB system studies with experimental and simulation studies, concerning mathematical modelling, system simulation, evaluation, capacity and operation strategy optimization. The system profitability, storage system choice and grid influence are also discussed, with the multi-objective optimization in large-scale systems with various participants expected as the future trend. In addition, several highlights of this topic are discussed in detail, including model predictive control, demand-side management, community energy storage system, peer-to-peer energy market scheme and mutual impacts on grid and system, emphasizing system flexibility, optimization complexity, large-scale simulation and the utility grid influences. Furthermore, some challenges and perspectives for future research are presented, with respect to DC distribution system revolution, large-scale vehicle to grid designs and impacts, multi-energy network operation and participation in the carbon transaction market. It is expected that this study could provide useful references and suggestions for researchers in this field of system design and power management of distributed solar PV.