参考资料：

1. Pytorch 并行训练（DP， DDP）的原理和应用 (opens new window)
2. Pytorch 中单机多卡分布式训练 (opens new window)
3. Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU & Distributed setups (opens new window)
4. DISTRIBUTEDDATAPARALLEL (opens new window)
5. Pytorch 分布式训练（DP/DDP） (opens new window)
6. DISTRIBUTED COMMUNICATION PACKAGE - TORCH.DISTRIBUTED (opens new window)
7. pytorch 多 gpu 并行训练 (opens new window)
8. PyTorch 源码解读之 DP & DDP：模型并行和分布式训练解析 (opens new window)