NCCL
NVIDIA's library for collective operations between GPUs — all-reduce, all-gather, broadcast — using the fastest link available. Every multi-GPU training framework sits on top of it.
A definition is not an understanding. This is taught properly in Collectives and the numbers that decide a design, part of Distributed training.
Learn this properlyTerms from the same topic