Distributed & Memory

is_ddp_available	Return True if torch.distributed is available on this system.
setup_ddp	Initialize the default distributed process group.
cleanup_ddp	Destroy the default distributed process group.
wrap_ddp	Wrap a MappingModel for DDP training.
profile_peak_memory	Measure the peak memory used by fn during its execution.
benchmark_strategies	Compare SLVT, layerwise, and grouped strategies on a given spec.