can sparse all-reduce keep efficiency with large number of gpu workers？ #1140
Description
in my opinion, when the gpu cluster scaled up to several hundred workers, high sparsification ratios still generate significant communication overheads, which even worst than DenseAllReduce.