A small team ships a developer tool in early 2024. It routes requests across GPT-4, Claude, and a fine-tuned open-source model, picks the cheapest option that meets the quality threshold, and logs everything for debugging. Customers love it. The team raises a seed round. Nine months later, OpenAI ships structured outputs natively, Anthropic releases context caching, and Google adds routing hints to the Gemini API. The team's core value proposition just became a feature in three different platform release notes.
This is not a story about a bad product. It is a story about where middleware lives in the AI stack and what happens when the platforms above and below it start moving.
What Middleware Actually Does
The AI stack has three rough layers. At the bottom, foundation model providers train and serve models: OpenAI, Anthropic, Google, Meta, Mistral. At the top, end-user applications consume AI capabilities to do something specific: answer customer questions, generate marketing copy, analyze contracts. Middleware sits between them.
Middleware handles orchestration, routing, prompt management, observability, evaluation, and cost control. Companies like LangChain, LlamaIndex, Weights and Biases, Helicone, and dozens of smaller players built businesses in this space. Some focus on developer tooling. Some focus on the ops layer. Some try to do both.
The value proposition is real. Calling one model directly is simple. Running a multi-step agent workflow across several models, with fallbacks, rate limit handling, output validation, and cost tracking, is not. Middleware reduces that complexity.
The problem is that the complexity middleware reduces is the same complexity platform providers are racing to eliminate themselves.
Where the Squeeze Comes From
Platform providers have two strong incentives to push upward into middleware territory. First, capturing more of the stack means capturing more revenue. A customer who orchestrates through a vendor's native tooling generates more API calls to that vendor. Second, every friction point a developer hits is a churn risk. Providers that own that layer keep more developers in their ecosystem.
This is not speculation. OpenAI's Assistants API handles thread management, file retrieval, and tool-calling natively. Anthropic's extended thinking and prompt caching reduce the need for external prompt optimization layers. Google's Vertex AI ships with built-in evaluation and model routing. Every one of those features lands directly on top of something a middleware company was charging for.
The squeeze from below comes from customers. Enterprises adopting AI at scale want fewer vendors, not more. A company running an AI-powered workflow does not want four separate contracts, four SLAs, and four support relationships for what should be one integrated system. Middleware vendors that cannot show clear, measurable value above what the platform provides are the first ones cut in a vendor consolidation.
The companies caught in the middle are those selling thin orchestration. If your product primarily wraps platform capabilities and adds a routing layer, the platforms have you in their roadmap.
Three Positions in the Squeeze
Not every middleware company is equally exposed. The level of risk depends on what the middleware actually owns.
Thin orchestrators are the most exposed. Their value is routing, load balancing, and cost optimization across models. As platform providers add native equivalents, the differentiation shrinks. These tools are useful, but they are not sticky. Switching costs are low. The revenue ceiling is low. Unless the team pivots, the category collapses into a commodity.
Workflow-depth players are in a stronger position. These are the middleware companies that have embedded themselves into specific industries or workflows. A tool that orchestrates legal document review, handles the specific citation and verification requirements of that domain, and integrates with the software a legal team already uses is not easily replaced by a generic routing layer. The platform can ship context caching; it cannot ship the legal workflow logic the middleware vendor spent two years building with actual firms. The switching cost is high. The value is measurable. The customer does not want to rebuild it.
Infrastructure and verification layers are a third position, growing more relevant as agents become more capable. Evaluation frameworks, output validation systems, compliance checks, and audit logging are not features a platform wants to own deeply. They are liabilities. A platform that certifies its own outputs is a conflict of interest. Middleware that provides independent verification, grounding checks, and traceable output histories is building something the platforms are motivated to leave outside their walls.
The Real Tradeoffs
Staying thin has one advantage: it is easy to build and easy to sell. The business has no moat, though. The moment a platform ships the feature you are wrapping, your renewal conversations get harder.
Going deep into workflows requires domain expertise the team may not have, longer sales cycles, and heavier implementation work. What you get is a defensible position. A legal workflow tool or a clinical documentation system that has passed procurement, security reviews, and user training at ten health systems is not being replaced because Anthropic shipped a new API feature. The switching cost is too high.
The infrastructure and verification play is technically demanding and harder to monetize in the short term. Enterprises buy it, but the sale is slower and the budget lives in compliance or legal, not engineering. The demand is structural, though. As agents make more autonomous decisions, someone has to verify the outputs. That need is not going away, and platforms have limited interest in owning it.
The one position that is not viable long term is staying in the middle of the middle, general-purpose orchestration with shallow customer relationships and no measurable workflow ownership. That space gets smaller every quarter.
Two Paths
If you are building in middleware today, you have two realistic options.
One is to go deep into one vertical and own the workflow layer. Pick a domain where AI is being deployed but where the specific requirements, compliance constraints, and integration complexity make generic tools insufficient. Build the workflow logic. Get into procurement cycles. Make yourself the thing customers would have to rebuild from scratch if they replaced you. This is slower, but it produces a real business.
The other is to build infrastructure that agents will need regardless of which models they use. Output verification, grounding against authoritative sources, audit trails, policy enforcement on agent actions. These are not feature requests; they are requirements showing up in enterprise procurement checklists right now. Building here means competing on reliability and compliance rather than feature sets, which is a more durable competition to be in.
What does not work is optimizing the middle layer and hoping the platforms stop shipping. They will not stop. The AI stack is still consolidating and every major provider is expanding upward. Middleware companies waiting for the squeeze to ease are reading the situation backwards.
The teams moving fastest into deep workflows or verification infrastructure are the ones who saw the platform trajectory early. They stopped defending territory the platforms were always going to take and moved somewhere the platforms cannot follow as easily.