LLMs like Claude are effective at categorizing unstructured 'open code' notes from error analysis into structured 'axial codes' or failure modes.

Coding agents are a special case for evaluation because the developers are also the domain experts and constantly dogfood the product, which shortens the feedback loop.

Major AI labs have historically focused on general benchmarks like MMLU and HumanEval, which often do not correlate with performance on product-specific quality dimensions.