LLMs like Claude are effective at categorizing unstructured 'open code' notes from error analysis into structured 'axial codes' or failure modes.

Coding agents are a special case for evaluation because the developers are also the domain experts and constantly dogfood the product, which shortens the feedback loop.

Major AI labs have historically focused on general benchmarks like MMLU and HumanEval, which often do not correlate with performance on product-specific quality dimensions.

Reviewing and analyzing application data is the highest ROI activity for improving an AI product.

Google Sheets' integrated AI, using models like Gemini, can automatically categorize free-form error notes into predefined failure categories based on a simple prompt.

Observability tools such as Braintrust, Phoenix Arise, and LangSmith are used for logging and analyzing LLM application traces.