Groq — LPU inference chips promising order-of-magnitude faster LLM serving
September 14 at 04:37 · $0.112 total
Investment Memo: Groq
Note: This memo treats Groq as a hypothetical Series A investment; the company is now later-stage.
Thesis
Groq is building an inference-only processor for large language models. Its LPU uses on-die SRAM and deterministic dataflow scheduling rather than HBM and caches. The result is order-of-magnitude lower latency: public demos showed hundreds of tokens per second on large open models when GPUs were doing tens. If that speed advantage becomes a sustained total-cost-of-ownership advantage, Groq could own the serving layer of the AI stack. Inference is likely to become larger than training as AI applications proliferate. A merchant silicon company that captures even a small share of that market could be worth $50–100B. From a hypothetical Series A entry, that is a fund-returner.
Product & Wedge
The product is the LPU / Tensor Streaming Processor, sold as cards, nodes, racks, and through GroqCloud. The architecture is deterministic: the compiler schedules data movement statically, avoiding cache misses and enabling high utilization at low batch sizes. It does not target training. The wedge is developer-facing LLM inference: GroqCloud offers API access to open models with latency low enough to change product behavior — real-time voice agents, coding assistants, agentic loops. This is a classic developer-adoption wedge: win workloads where speed is the gating constraint, then expand to standard serving.
Market & Competition
The addressable market is AI inference, which could exceed $100B annually by 2030. Competition is severe and real:
- NVIDIA: H100/H200/B200, TensorRT-LLM, CUDA, NVLink, and system-level distribution; the default.
- AMD: MI300X/ROCm, improving price-performance.
- Hyperscaler custom silicon: Google TPU v5e/v5p, AWS Trainium/Inferentia, Microsoft Maia, Meta MTIA.
- Startups: Cerebras, SambaNova, d-Matrix, Tenstorrent, Etched.
Groq is a merchant silicon vendor in a market where cloud providers are vertically integrating. It must either win sockets at large enterprises/clouds or build a successful cloud itself.
Traction & Business Signal
Publicly known traction is early but real. Groq went viral for serving Llama/Mixtral models at hundreds of tokens per second. GroqCloud has publicly reported 400k+ developer sign-ups and 19k+ applications built as of mid-2024. The company raised a $640M Series D at $2.8B led by BlackRock with strategic participation from Cisco and Samsung, signaling manufacturing and enterprise support. Revenue, gross margin, utilization, paying customer count, churn, and cost per token are unknown. Developer enthusiasm is a positive signal but not yet proof of a durable business.
Risks
Three risks could kill this:
- NVIDIA ecosystem lock-in and TCO compression. Even if Groq has lower latency, NVIDIA can narrow the gap through software, speculative decoding, and newer GPUs, while bundling inference with training and deployment. Buyers may not switch because CUDA, staffing, and cloud credits are entrenched. Speed alone may not overcome switching costs.
- Hardware unit economics. SRAM is expensive per bit and consumes die area. Groq chips have limited memory capacity, so large models require many chips. That can inflate capex, power, and hardware cost per token. If GroqCloud’s low pricing is subsidized by venture capital, the business may not be sustainable at scale.
- Squeeze between NVIDIA and hyperscaler ASICs. Google, AWS, Microsoft, and Meta have their own inference silicon tailored to their workloads. They are unlikely to buy much from Groq. Meanwhile NVIDIA owns the merchant market. Groq could become a fast but orphaned architecture with too few customers to amortize chip development.
Verdict
Pass at a hypothetical Series A. The technology is genuinely differentiated and the team has TPU pedigree, but the business risk is binary and public data does not yet prove sustainable cost advantage. The one question I would need answered first: Can Groq show a third-party, realistically loaded TCO model where LPU serving a frontier-scale open model beats NVIDIA GPU serving by at least 2–3x on cost per million tokens, after depreciation, power, utilization, and migration? If yes, I would re-underwrite aggressively; if no, speed is a feature, not a company.
Investment memo: Groq (hypothetical Series A)
Thesis. Inference, not training, will dominate AI compute spend as models proliferate into products. Groq’s LPU architecture targets a 10x+ step-function in tokens/sec and $/token versus GPU clusters for decoder-heavy LLM serving. If they own even a mid-single-digit share of a multi-hundred-billion-dollar inference TAM by the late 2020s—while remaining the latency/cost leader for real-time agents, voice, and coding copilots—this is a fund-returner. The bet is that a purpose-built, compiler-first chip plus a simple cloud API can unseat NVIDIA’s general-purpose monopoly in the serving layer before the software moat becomes unbreakable.
Product & wedge. Groq’s Language Processing Unit is a deterministic, software-defined architecture (SRAM-heavy, no caches or speculative execution) paired with a compiler that maps entire models onto a known execution graph. Public claims: hundreds of tokens/sec on Llama-class models at low latency and high energy efficiency versus H100-class GPUs. The wedge is not “better training” but serving: predictable low latency, high throughput, and simpler ops for production inference. GroqCloud (public API) is the go-to-market: developers get speed without buying racks. Hardware sales and on-prem remain secondary until software and compiler maturity are proven at scale. Differentiation lives in the compiler + deterministic silicon, not in raw FLOPS.
Market & competition. LLM inference demand is exploding (chat, agents, RAG, real-time voice/video). TAM is large and growing; Groq does not need to win training. Real competitors: NVIDIA (CUDA + H100/B200 ecosystem, software lock-in, data-center incumbency); Cerebras (wafer-scale, high throughput, different software stack); SambaNova (reconfigurable dataflow, enterprise focus); AMD (MI300, catching up on software); hyperscaler ASICs (Google TPU, AWS Inferentia/Trainium, custom silicon); plus Tenstorrent and others. Groq’s public positioning is speed and simplicity versus GPU clusters, not versus every ASIC. The risk is that NVIDIA’s software + volume + customer inertia absorbs most of the serving market even if Groq is faster on paper.
Traction & business signal. Publicly known: GroqCloud launched with Llama and other open models; high-profile demos of very high tokens/sec; founder Jonathan Ross (ex-Google TPU); later-stage capital raises (hundreds of millions reported in 2024 at multi-billion valuation—exact Series A terms unknown). Customer logos, ARR, gross margins, chip yield, and production volume at scale: unknown. No public evidence of large, sticky enterprise contracts or hyperscaler design wins at the time of this memo. Signal is technical credibility and developer interest, not yet proven unit economics or repeatable sales.
Risks (the three that kill the deal).
- NVIDIA software + ecosystem lock-in. CUDA, TensorRT, and the existing MLOps stack are the real moat. Even a 5–10x hardware advantage can lose if customers will not rewrite serving pipelines or if NVIDIA matches “good enough” latency via software and next-gen GPUs. Groq must win on compiler ease and cloud simplicity faster than NVIDIA iterates.
- Capital intensity and manufacturing execution. Custom silicon at volume requires sustained hundreds of millions in capex, foundry access, and yield. A miss on process node, packaging, or supply chain (or a competitor’s volume discount) can destroy unit economics before software wins. Unknown whether Groq can scale beyond boutique volumes without becoming a capital sink.
- Inference commoditization and hyperscaler vertical integration. If open models + NVIDIA/AMD/hyperscaler ASICs drive $/token down fast, Groq’s premium evaporates. Hyperscalers may never buy Groq silicon and will push their own inference stacks. Groq then becomes a niche cloud API with limited pricing power.
Verdict: Pass at hypothetical Series A, pending one answer. At Series A the company would still be pre-scale silicon and pre-repeatable revenue. The technical story is compelling and the TAM is real, but the three risks above are existential and the public traction is still demo- and cloud-API-level, not proven economics. I would pass unless the one question is answered first: What is the current and 12-month forward gross margin and volume on GroqCloud plus any committed hardware/on-prem deals, and what is the compiler/software switching cost versus NVIDIA for a typical production customer? Without evidence that they can manufacture at acceptable yield and that customers will actually switch serving stacks, this remains a high-upside science project, not a Series A investment. Later-stage rounds can re-underwrite once those numbers exist.