Traditional DRAM-based main memory systems face several challenges with memory refresh overhead, high latency, and low throughput as the industry moves towards smaller DRAM cells. These issues have been exacerbated by the emergence of data-intensive applications in recent years. Memories based on phase change materials (PCMs) offer promising solutions to these challenges. PCMs store data in the material's phase, which can shift between amorphous and crystalline states when external thermal energy is supplied. This is often achieved using electrical pulses. Alternatively, using laser pulses and integration with silicon photonics offers a unique opportunity to realize high-bandwidth and low-latency photonic memories. But to realize photonic memories, several challenges that are unique to the photonic domain such as crosstalk, optical loss management, and laser power overhead must be addressed. In this work, we present COMET, the first cross-layer optimized optical main memory architecture that uses PCMs. In architecting COMET, we explore how to use silicon photonics technology and PCMs together to design a large-scale main memory system while exploring related challenges and proposing solutions at the PCM cell, photonic memory circuit, and memory architecture levels. Based on our evaluations, COMET offers 5.l× better bandwidth (BW), 12.9× lower energy-per-bit (EPB), and 65.8x better BW/EPB than the best-known prior work on photonic main memory architecture design.