ddp for gradient checkpointing - SERP Analysis

https://discuss.pytorch.org/t/ddp-and-gradient-checkpointing/132244

https://discuss.huggingface.co/t/ddp-gradient-checkpoint-crashes/58432

https://discuss.pytorch.org/t/ddp-with-gradient-checkpointinting-confusing-documentation/201722

https://github.com/allenai/longformer/issues/63

https://www.kaggle.com/code/wouldyoujustfocus/increase-batchsize-with-gradient-checkpoint-ddp-dp

https://lightning.ai/forums/t/gradient-checkpointing-ddp-nan/398

https://huggingface.co/docs/accelerate/en/concept_guides/gradient_synchronization

https://stackoverflow.com/questions/70386800/what-is-the-proper-way-to-checkpoint-during-training-when-using-distributed-data

https://medium.com/%40epappas/scaling-machine-learning-distributed-data-parallel-ddp-training-with-ray-and-pytorch-f445cf83f588

https://lightning.ai/forums/t/gradient-checkpointing-ddp-nan/398/5

https://medium.com/pytorch/how-activation-checkpointing-enables-scaling-up-training-deep-learning-models-7a93ae01ff2d

https://www.reddit.com/r/MachineLearning/comments/wxvlcc/d_does_gradient_accumulation_achieve_anything/

https://pytorch-lightning.readthedocs.io/en/1.3.8/advanced/advanced_gpu.html

https://stackoverflow.com/questions/68000761/pytorch-ddp-finding-the-cause-of-expected-to-mark-a-variable-ready-only-once

https://wandb.ai/patrickvonplaten/huggingface/reports/Wav2Vec2-DistributedDataParallel-DDP-8-GPU-V100--VmlldzoxMDQwMDU3

https://www.reddit.com/user/TrelisResearch/comments/1c0rsmk/multigpu_finetuning_with_ddp_and_fsdp/

https://www.kaggle.com/code/danielhanchen/hf-sdpa-oasst-bsz1-t4-ddp

https://www.youtube.com/watch?v=gXDsVcY8TXQ

https://jordano-jackson.tistory.com/172

https://www.linkedin.com/posts/aleksagordic_over-the-past-period-i-had-an-immense-privilege-activity-7217531755501633536-XpTI

https://learnopencv.com/distributed-parallel-training-pytorch-multi-gpu-setup/

https://www.gilesthomas.com/2024/09/fine-tuning-9

https://github.com/Deci-AI/super-gradients/actions/runs/8587553507

https://apxml.com/courses/how-to-build-a-large-language-model/chapter-19-checkpointing-fault-tolerance/handling-distributed-checkpointing

https://pytorch-cn.com/tutorials/intermediate/ddp_tutorial.html

https://hichoe95.tistory.com/124

https://saturncloud.io/blog/how-to-solve-cuda-out-of-memory-in-pytorch/

https://colossalai.org/docs/basics/booster_plugins/

https://www.union.ai/blog-post/fine-tune-llama-2-with-limited-resources

https://massedcompute.com/faq-answers/?question=Can%20you%20explain%20the%20difference%20between%20PyTorch%27s%20DataParallel%20and%20DistributedDataParallel%20for%20memory%20optimization?

https://auto.gluon.ai/stable/tutorials/multimodal/advanced_topics/customization.html

https://dongdongbh.tech/ddp/

https://pytorch-lightning.readthedocs.io/en/1.1.8/training_tricks.html

https://apxml.com/courses/advanced-pytorch/chapter-5-distributed-training-parallelism/practice-ddp-script

https://docs.axolotl.ai/docs/config.html

https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html

https://velog.io/%40jminj/Pytorch%EC%97%90%EC%84%9C-memory%EC%82%AC%EC%9A%A9-Guide

https://towardsdatascience.com/a-comprehensive-guide-of-distributed-data-parallel-ddp-2bb1d8b5edfb/

https://colossalai.org/docs/features/zero_with_chunk/

https://www.youtube.com/watch?v=5B4d0FuxSQc

https://siboehm.com/articles/22/data-parallel-training

https://zhuanlan.zhihu.com/p/448395808

https://arxiv.org/pdf/2304.11277

https://www.linkedin.com/posts/deshwalmahesh_cuda-gpu-nvidia-activity-7207245376813191169-9APO

https://engineering.fb.com/2021/07/15/open-source/fsdp/

https://www.youtube.com/watch?v=1XuibaVRewc

https://aman.ai/primers/ai/grad-accum-checkpoint/

https://tutorials.pytorch.kr/beginner/ddp_series_multigpu.html

https://jlin27.github.io/intermediate/ddp_tutorial.html

https://en.wikipedia.org/wiki/DeepSeek

https://docs.habana.ai/en/latest/PyTorch/PyTorch_Scaling_Guide/DDP_Based_Scaling.html

https://docs.jax.dev/en/latest/gradient-checkpointing.html

https://muellerzr.github.io/blog/gradient_accumulation.html

https://python-bloggers.com/2024/09/mastering-gradient-checkpoints-in-pytorch-a-comprehensive-guide/

https://www.youtube.com/watch?v=ldBmHNva_Fw