Where is enterprise data infrastructure spend consolidating versus fragmenting as LLMs become a primary query interface?

As Large Language Models (LLMs) become a primary enterprise interface, a significant divergence is occurring in data infrastructure spending, with consolidation at the foundational data and compute layers and fragmentation in specialized tooling. The core enterprise strategy is to consolidate proprietary information into unified data platforms like Snowflake, Databricks, or Palantir [5, 11]. This creates a defensible moat, as the foundational LLMs themselves are widely seen as a commodity where performance has converged and price is the main differentiator [1, 3, 12]. Consequently, enterprises are expected to adopt a multi-model approach, using gateways to manage access to various LLMs based on cost and specific task requirements . This dynamic concentrates value and spending in the proprietary data layer, which is considered one of the most durable components of the new AI stack, while commoditizing the model layer itself [20, 24].

At the lowest level of the stack, spending is consolidating heavily into the hands of hyperscalers like Google and AWS, who are the primary beneficiaries of an unprecedented **"CapEx arms race"** . These cloud providers capture value by selling massive amounts of compute to LLM companies and then reselling AI services and infrastructure to enterprise customers, creating a symbiotic relationship that solidifies their role as a central chokepoint [4, 14]. This intense investment in compute infrastructure is consuming the majority of their free cash flow and is fueled by the intense, fluid competition between foundation models [4, 15]. While this represents a clear consolidation of capital, some infrastructure-focused SaaS companies are also benefiting by providing essential "plumbing" for AI applications and agents .

Between the consolidated data platforms and the hyperscaler compute layer, a fragmented ecosystem of specialized infrastructure is emerging to address new AI-specific workloads. The need for Retrieval-Augmented Generation (RAG) persists, as large context windows are not a panacea for connecting private datasets due to issues of cost, scale, and poor recall [10, 27]. This has created a vibrant, fragmented market for vector databases, with new architectures based on object storage emerging to make petabyte-scale vector search economically viable [19, 27]. Similarly, specialized high-performance databases like ClickHouse are positioning themselves as "picks and shovels" for AI leaders, winning adoption through developer-centric, product-led growth rather than traditional enterprise sales [9, 30]. This fragmentation reflects a need for new tools to solve specific problems like incremental indexing and filtered search in production AI systems .

This restructuring of the data stack is driven by the fundamental shift toward AI agents becoming the primary consumers of infrastructure, surpassing human developers in traffic volume for some services [6, 25]. This forces infrastructure providers to optimize for programmatic, agent-driven consumption [9, 25]. This paradigm reinforces the trend toward data consolidation, as it is more efficient for an agent to query a single unified data warehouse than to connect to dozens of disparate systems . While the unit cost per token is decreasing, this rise in complex, agent-driven queries is causing aggregate enterprise spending on AI to increase . The entire infrastructure landscape is therefore reorienting around a future where machine-to-machine interaction defines demand, driving both consolidation in core data repositories and fragmentation in the specialized tools that enable agentic workflows.