Distributed Deep Learning (DDL), as a paradigm, dictates the use of GPU-based clusters as
the optimal infrastructure for training large-scale Deep Neural Networks (DNNs). However …