Blog – Vinay R Jumani

Scaling ML Models: Data, Tensor, and Pipeline Parallelism Explained

July 23, 2026 Distributed Training, LLMs, Systems, GPU 18 min read

A comprehensive breakdown of how LLMs scale across GPUs: from memory allocation fundamentals (weights, gradients, optimizer states) to data parallelism, Megatron-LM tensor parallelism, GPipe pipeline parallelism, activation checkpointing, mixed precision, and DeepSpeed ZeRO.

Profiling CUDA GEMM Kernels: 2D Block Tiling vs Vectorized Loads

June 7, 2026 CUDA, Systems, HPC, Profiling 20 min read

In this post, I profile and compare a 2D block-tiled GEMM kernel against a vectorized variant on an NVIDIA RTX A6000 GPU, analyzing memory coalescing, shared memory bank conflicts, and occupancy using Nsight Systems and Nsight Compute.

Scaling ML Models: Data, Tensor, and Pipeline Parallelism Explained

Profiling CUDA GEMM Kernels: 2D Block Tiling vs Vectorized Loads

Subscribe to the Mailing List