Groq — LPU inference chips promising order-of-magnitude faster LLM serving
September 14 at 04:21 · $0.121 total
Investment Memo: Groq
Stage framing: Hypothetical Series A · Sector: AI inference silicon + cloud serving
---
Thesis
The AI compute market is bifurcating: training is a capex arms race NVIDIA has effectively won, but inference is where the recurring, volume economics live — every deployed LLM query, forever. If inference is 80–90% of lifetime AI compute spend (a widely held industry view), the market supports multiple $10B+ outcomes. Groq's bet is that a deterministic, software-scheduled architecture (the LPU — "Language Processing Unit") built specifically for sequential token generation can deliver order-of-magnitude latency advantages over GPUs, which were designed for parallel throughput, not low-latency autoregressive decoding.
The fund-returner case: latency becomes a product feature, not an infrastructure detail. Voice agents, real-time copilots, agentic loops that chain dozens of LLM calls — all are latency-bound. If Groq owns the "fast inference" category the way Snowflake owned "cloud data warehouse," this is a $50B+ outcome. Founder Jonathan Ross led the team that invented Google's TPU — rare, credible founder-market fit in silicon.
Product & Wedge
The LPU is a deterministic tensor streaming processor: no HBM (it uses on-chip SRAM), no dynamic scheduling, compiler-controlled execution. This yields predictable, very low latency per token — publicly demonstrated at 300–800+ tokens/sec on Llama-class models, visibly faster than GPU-served endpoints.
The wedge is smart: GroqCloud, an API-first serving business for open-weight models (Llama, Mixtral, Whisper), not selling chips. This sidesteps the classic chip-startup death trap (convincing hyperscalers to qualify new silicon) and converts hardware advantage into a developer-facing product with OpenAI-compatible APIs. Developers switch endpoints in one line of code. Groq has claimed hundreds of thousands of registered developers on GroqCloud.
Caveat: SRAM-only design means small memory per chip (~230MB), so serving a large model requires hundreds of chips racked together. The architecture trades capital intensity for latency.
Market & Competition
Inference serving TAM plausibly reaches $100B+ by 2030. Competition is brutal and comes from four directions:
- NVIDIA — the default. H100/B200 plus TensorRT-LLM keeps closing the latency gap in software; ecosystem lock-in via CUDA is the moat Groq must route around.
- Direct architectural rivals: Cerebras (wafer-scale, also posting extreme tokens/sec benchmarks and running an inference cloud — the closest comp), SambaNova, and to a lesser degree Tenstorrent and d-Matrix.
- Hyperscaler silicon: AWS Inferentia/Trainium, Google TPU, Microsoft Maia — vertically integrated, subsidized, and capturing the largest inference workloads internally.
- Inference-serving software layers: Together AI, Fireworks, Baseten — competing for the same developer API dollar on commodity GPUs, often at lower prices.
Groq's differentiation is real but narrow: latency leadership. Cerebras contests even that claim head-on.
Traction & Business Signal (publicly known)
- Funding/valuation: $640M Series D (Aug 2024) led by BlackRock at ~$2.8B valuation; subsequent 2025 reporting of a raise at ~$6–7B. Cumulative funding well over $1B.
- Developers: Groq has publicly claimed 1M+ registered GroqCloud developer accounts (self-reported; conversion to paid unknown).
- Deals: Announced ~$1.5B commitment tied to Saudi Arabia (Aramco Digital / HUMAIN partnership, Dammam datacenter) — a genuinely large sovereign anchor, though concentration cuts both ways. Partnerships announced with Meta (Llama API serving) and Bell Canada.
- Revenue: Not disclosed. Press reports suggest revenue materially below earlier projections; actual ARR, gross margin, and utilization: unknown. Chip unit economics at scale: unknown.
Risks (the three that kill it)
- Unit economics of the SRAM architecture. Serving one 70B model reportedly requires ~500+ LPU chips versus a handful of H100s. Even at lower per-chip cost, capex per token served may never beat GPUs at scale, especially as NVIDIA's Blackwell improves inference perf/$ by multiples. If Groq's speed comes at structurally negative or thin gross margins, the cloud business is a subsidized demo, not a company. This is the existential question.
- Latency advantage is perishable. Speculative decoding, better batching, custom GPU kernels, and Blackwell's NVLink-scale inference are compressing the gap. Cerebras already contests the crown. If "fast enough" GPU inference arrives at commodity price, Groq's category evaporates and it's a subscale chip vendor without a CUDA-equivalent ecosystem.
- Capital intensity + customer concentration. Building datacenters full of proprietary silicon requires billions; the Saudi deal supplies capital but concentrates revenue and geopolitical exposure in one relationship. A financing winter or a soured sovereign relationship strands the balance sheet.
Verdict
Invest — with conditions. At a hypothetical Series A (i.e., pre the current mega-rounds), this is a bet worth making: TPU-pedigree founder, real architectural differentiation, and a wedge (API-first) that avoids the historical chip-startup graveyard. Asymmetry favors the bull case; inference is the right layer.
The one question to answer first: What is the fully loaded cost per million tokens served on GroqCloud — including chip amortization — versus an H100/B200 deployment at equivalent quality, and what is the gross margin at current pricing? If the answer is "negative and structurally so," pass. Speed that loses money per token is a benchmark, not a business.
Investment memo: Groq (hypothetical Series A)
Thesis. Inference, not training, is where LLM economics will concentrate: every query, every agent loop, every real-time product. NVIDIA owns training via CUDA and scale; inference is still relatively open on latency, $/token, and power. Groq’s LPU architecture (deterministic, compiler-first, SRAM-heavy) has publicly demonstrated order-of-magnitude lower latency and higher tokens/sec on Llama-class models versus GPU baselines in demos. If that advantage holds at production scale and cost, Groq can become the default serving layer for latency-sensitive applications (agents, voice, coding copilots, search). A category winner here can return a fund: inference TAM is large, recurring, and expanding with model size and usage. The bet is that a purpose-built inference chip plus a tight compiler can outrun general-purpose GPUs on the metric that matters for serving—time-to-first-token and sustained throughput—before NVIDIA and hyperscalers close the gap.
Product & wedge. Groq sells LPU chips and GroqCloud inference. The wedge is extreme speed and predictability: public demos have shown hundreds of tokens/sec on 70B-class models with low jitter, enabled by a software-defined hardware approach (the compiler maps the model onto a deterministic datapath rather than relying on GPU scheduling). Positioning is “fastest LLM serving,” not general training. Cloud API is the go-to-market: developers try GroqCloud, then (in theory) pull chips or dedicated capacity. Differentiation is architectural (LPU vs GPU/TPU) plus compiler, not just another CUDA clone.
Market & competition. LLM inference is already a multi-billion-dollar run-rate market and growing with usage. Competitors are real and well-capitalized: NVIDIA (H100/H200/B200 + TensorRT-LLM; software and ecosystem moat); AMD (MI300 and ROCm, catching up on inference); Cerebras (wafer-scale, also chasing inference speed); SambaNova; Tenstorrent; hyperscaler ASICs (Google TPU, AWS Inferentia/Trainium, custom Microsoft/Meta silicon). Software stacks (vLLM, TensorRT-LLM, SGLang) and model quantization further commoditize “fast enough” serving on GPUs. Groq must win on a combination of latency, cost, and ease of use that GPUs plus software cannot match for the highest-value workloads.
Traction & business signal. Publicly known: high-profile speed demos and GroqCloud availability; founder pedigree (Jonathan Ross, Google TPU). Specific revenue, ARR, customer logos at scale, chip shipments, gross margins, and retention are unknown. Cloud usage and waitlists have been discussed in press, but audited commercial traction at Series A vintage is not public. Treat commercial proof as unproven in this memo.
Risks (the three that kill the deal).
- NVIDIA + software close the gap. TensorRT-LLM, speculative decoding, better kernels, and next-gen GPUs (plus CUDA lock-in) can make “good enough” latency cheap enough that Groq’s architectural edge becomes a rounding error. Inference buyers optimize TCO and ecosystem, not peak demo tokens/sec.
- Compiler and generality. LPU performance depends on the compiler mapping models efficiently. If new architectures (MoE, long context, multimodal, custom kernels) require constant compiler work or leave performance on the table, Groq becomes a niche for a few Llama-like models rather than a platform. Unknown: how much of the speed is model-specific vs. durable.
- Capital and go-to-market. Custom silicon is capital-intensive; cloud inference is a race to the bottom on price unless there is a durable cost or latency moat. Hyperscalers can subsidize inference and bundle it. Without clear unit economics and a path from GroqCloud trials to committed capacity or chip sales, Groq burns cash competing with NVIDIA’s volume and AWS/GCP/Azure distribution. Customer concentration and switching costs are unknown.
Verdict: Pass at hypothetical Series A, pending one answer.
The architecture and demos are exciting; the market is real. At Series A, the deal dies on unproven commercial pull and the three risks above. I would need a clear answer to: What is the measured, production TCO (latency, $/million tokens, power, and engineering hours) versus NVIDIA + TensorRT-LLM / vLLM on the same models and SLAs for at least two paying customers over 90+ days—and how much of that gap is compiler/LPU vs. software that GPUs can copy? Without that, this is a science project with a famous founder, not a fund-returner. If that gap is large, durable, and already converting to revenue, the thesis reopens. Until then, pass.