Gspmd: general and scalable parallelization for ml computation graphs
We present GSPMD, an automatic, compiler-based parallelization system for common
machine learning computations. It allows users to write programs in the same way as for a
single device, then give hints through a few annotations on how to distribute tensors, based
on which GSPMD will parallelize the computation. Its representation of partitioning is simple
yet general, allowing it to express different or mixed paradigms of parallelism on a wide
variety of models. GSPMD infers the partitioning for every operator based on limited user …
machine learning computations. It allows users to write programs in the same way as for a
single device, then give hints through a few annotations on how to distribute tensors, based
on which GSPMD will parallelize the computation. Its representation of partitioning is simple
yet general, allowing it to express different or mixed paradigms of parallelism on a wide
variety of models. GSPMD infers the partitioning for every operator based on limited user …
以上显示的是最相近的搜索结果。 查看全部搜索结果