Keep pulling the thread on Tri Dao.
CODA is a GPU kernel abstraction that expresses memory-bound Transformer computations as GEMM-plus-epilogue programs.
The CODA abstraction allows many Transformer operators to be executed on-chip using a GEMM output tile before it is written to global memory.
The CODA interface can cover nearly all non-attention computations in both the forward and backward passes of a standard Transformer block.
CODA kernels, whether authored by humans or Large Language Models, achieve high performance on representative Transformer workloads.
The CODA abstraction provides a fixed GEMM mainloop and a set of composable epilogue primitives for scaling, reductions, pairwise transformations, and accumulation.
CODA's design preserves the performance structure of expert-written GEMM kernels.
In Transformer training systems, a significant portion of the total execution time is consumed by memory-bound operators rather than just dense linear algebra computations.
Operations such as normalization, activations, and residual updates in Transformer training create a data movement bottleneck by repeatedly moving large intermediate tensors through global memory.