LLM Inference Infrastructure Market Map
(1) Thesis
Inference is being restructured from "rent a GPU and run vLLM" into a layered value chain where margin migrates away from raw compute toward the routing, optimization, and workload-aware serving layers that arbitrage the widening gap between hardware cost and per-token pricing.
---
(2) Segments
A. Serverless Inference Clouds (GPU-abstracted, per-token/per-second)
Companies: Together AI, Fireworks AI, Baseten, Replicate, Modal, Fal (media-focused) Dynamics: Brutal price competition on open-weight models (Llama, DeepSeek); differentiation shifting to proprietary kernels, speculative decoding, and enterprise features (VPC deployment, compliance) as per-token prices race toward marginal cost.
B. Neo-clouds / GPU Capacity Providers
Companies: CoreWeave, Lambda, Crusoe, Nebius, Vast.ai, SF Compute Dynamics: Capital-intensive, hyperscaler-squeezed; winners are locking in long-term anchor tenants (CoreWeave–Microsoft/OpenAI) while spot marketplaces commoditize the long tail. Value increasingly captured via financing structures, not technology.
C. Inference Optimization & Serving Software
Companies: vLLM (open source, backed by Anyscale ecosystem), NVIDIA (TensorRT-LLM/Dynamo), SGLang (open source), Anyscale, Neural Magic (acquired by Red Hat), Predibase (acquired by Rubrik) Dynamics: Open source dominates the serving engine layer, making pure-software monetization hard — hence the acquisition wave. Value accrues to whoever bundles it (NVIDIA, clouds) rather than standalone vendors.
D. Routing, Gateways & Model Orchestration
Companies: OpenRouter, LiteLLM, Portkey, Kong (AI Gateway), Martian (less sure of current traction), Not Diamond (less sure — small) Dynamics: Thin-margin aggregation today, but strategically positioned as the "control plane" — owns customer relationship, usage data, and failover logic across providers. Low switching costs cut both ways.
E. Custom Inference Silicon
Companies: Groq, Cerebras, SambaNova, Etched (pre-revenue, less sure), Tenstorrent Dynamics: Speed/cost advantages are real (Groq, Cerebras token throughput), but they compete against NVIDIA's software moat and must run their own clouds to monetize — turning chip companies into capital-hungry inference providers.
---
(3) Where value concentrates
Segment A (serverless inference clouds) wins the most venture-capturable value. They sit at the point of maximum arbitrage: they buy commoditizing compute (B), absorb free optimization software (C), and sell metered tokens with sticky enterprise attach (fine-tuning, evals, compliance). B is a real-estate/financing business, C has been structurally commoditized by open source, D is high-leverage but tiny revenue today, and E faces NVIDIA. Fireworks/Together/Baseten valuations reflect this. The caveat: hyperscalers and model labs (OpenAI, Anthropic first-party APIs) compress this layer over time, so winners must move up-stack into agent/workload infrastructure.
---
(4) White space
- Inference cost observability & FinOps for tokens — a "Datadog/CloudHealth for inference": cross-provider cost attribution per feature/agent/customer, cache-hit economics, prefill-vs-decode cost breakdown, and automated model-downgrade recommendations. Gateways log requests but nobody owns the CFO-facing optimization layer.
- Stateful/long-context inference infrastructure for agents — KV-cache persistence, sharing, and tiering as a service (cache-as-storage across sessions and providers), so multi-hour agent workloads don't re-pay prefill costs. Serving engines handle this per-node; no one offers it as a portable, cross-provider primitive. (Some labs and Moonshot/DeepSeek do context caching internally — but no neutral infra player owns it.)