Philip Kelly's book, "Inference Engineering," posits that AI inference is a full-stack problem combining CUDA, on-GPU optimization, and distributed systems.

The GPT OSS 120B model is considered too large for many use cases, making smaller models like the Gemma family more suitable for fine-tuning.

Baseten was a day-zero support partner for the launch of Google's Gemma 4 model.

Baseten operates a multi-region deployment that pools GPUs from America and Europe into a unified compute pool to minimize user latency.

Baseten was one of the first companies to use NVIDIA Dynamo for inference in a production environment.

The Gemma family of models from Google supports native image inputs, which is beneficial for enterprise use cases like KYC and document extraction.

The low-latency networking of Google Kubernetes Engine (GKE) saves Baseten a couple dozen milliseconds per turn between models in multi-model agentic systems.

Baseten utilizes Google Kubernetes Engine (GKE) to build its systems on top of Google Cloud GPUs.

The Gemma family of models provides a wide range of sizes, from 2 billion to 30 billion parameters.

Baseten offers NVIDIA's Nemotron models, such as Nemo Tron Super, through a pay-per-token API.

Migrating an inference system from NVIDIA's Hopper architecture to the Blackwell architecture requires significant software and kernel layer work.