How to make your models fast

CUDA kernel worklogs for understanding inference performance ( | GEMV)

A slowly growing book about reasoning through CUDA kernels from first principles.

I am learning how to make model inference faster by starting at the lowest level I can reasonably reach: individual CUDA kernels.

This book is a slowly growing compilation of those worklogs. For now, its scope is deliberately narrow. We write one kernel, count its computation and memory traffic, inspect how its threads use the GPU, and then improve it one bottleneck at a time.

This is not meant to be a broad guide to every layer of model scaling. There is no large roadmap that I am promising to fill in. I will add a chapter when I have learned enough to explain the problem honestly and work through an implementation.

How each worklog proceeds

Every chapter follows roughly the same loop:

1. Write the most straightforward implementation.
2. Count its FLOPs, memory traffic, and arithmetic intensity.
3. Look at what a warp actually computes and reads.
4. Change one part of the algorithm or its mapping to the GPU.
5. Recalculate the cost and write the improved kernel.

The intermediate versions stay in the chapter. The point is not only to arrive at fast code, but to make the reasoning that led there reusable.

Current chapters

Chapter 1 · Kernels GEMV One matrix, one vector, and an access pattern that quietly wastes most of the GPU’s memory bandwidth. Read the chapter →

Chapter 2 · Kernels Softmax From a direct row-wise implementation to online statistics, coalesced reads, and warp reductions. Read the chapter →

Chapter 3 · Kernels LayerNorm From three global-memory reads to coalesced Welford statistics and a shared-memory row cache. Read the chapter →

Chapter 4 · Kernels GEMM From one output per thread to shared-memory tiles, register micro-tiles, and vectorized loads. Read the chapter →