Every AI platform review I have sat in this year hits the same slide: “should we still be paying per token?” The math finally has real answers. As of mid-2026, a 70B-class open model served from an 8x H100 pod lands in the range of $1–$2.50 per million tokens at healthy utilization — FP8 quantization pushes the low end near $1 — while frontier API list prices for comparable quality run anywhere from $0.10 to $15 per million depending on tier. Those two ranges overlap, and that overlap is exactly why so many teams get this decision wrong.
This brief gives you the break-even bands, the honest per-million numbers on both sides, and the costs that never make it into the spreadsheet. Short version: volume and utilization decide this, not ideology.
The break-even numbers
Here is the framework I give every architecture team that asks. Treat the boundaries as bands, not lines — caching behavior, model class, and traffic shape move them — but the bands themselves have held up across every deployment I have reviewed this year.
| Monthly token volume | Cheapest option, as of mid-2026 | Why | |
|---|---|---|---|
| Low | Under ~20M tokens | Managed API | Idle GPU hours dominate; even one rented H100 at ~$3/hr is money burned at this volume |
| Gray zone | ~20M–100M tokens | It depends | Prompt caching, batch discounts, and whether a small model suffices decide the winner |
| High | 100M+ tokens, sustained | Self-hosted or dedicated rented GPUs | $1–$2.50/M on an 8x H100 pod undercuts frontier list pricing — if utilization stays high |
The single most important variable is utilization. At roughly 10% utilization, your real cost per token can run near 10x the headline rate — an idle H100 is more expensive per token than a premium frontier API. Self-hosting wins on volume you actually serve, not volume you project in a planning deck.
Where the self-hosting math comes from
The self-hosting story in 2026 is still overwhelmingly an NVIDIA story. Cloud H100 pricing has stabilized around $2.85–$3.50 per hour at most providers after falling hard from the 2024 peaks, and the software side — TensorRT-LLM, the NIM inference microservices, and community stacks like vLLM running on NVIDIA silicon — is where the per-token gains actually come from. FP8 quantization on Hopper-class hardware is the difference between roughly $4.50 per million tokens for a 70B model and something closer to $1–$1.70. That is NVIDIA’s real strength here: the optimization ecosystem is mature and the talent pool knows it.
The weakness is equally clear: you are buying into a supply-constrained, premium-priced platform, and the depreciation math is brutal. Blackwell-class hardware makes a pod you bought eighteen months ago look slow, and resale values reflect it. Whether you should own the metal at all is a separate decision — our GPU buy-vs-rent break-even analysis covers that math, and for teams going on-prem, the full cluster cost guide itemizes the power, cooling, and networking lines that double the sticker price.
Takeaway for the meeting: self-hosting a 70B-class model at $1–$2.50 per million tokens is real, but only above sustained high utilization on hardware someone still has to buy, rack, or reserve.
The API counterargument: prices mostly fell
The strongest argument against self-hosting is that the API vendors keep cutting the ground out from under your business case. Since mid-2025, effective per-token prices across the major providers have fallen on the order of 40–60% for comparable capability — through cheaper tiers, aggressive caching discounts, and batch pricing.
| Indicative list price (per M in / out) | Where it fits | |
|---|---|---|
| Google Gemini 2.5 Flash-Lite | ~$0.10 / $0.40 | High-volume classification, extraction, routing |
| Google Gemini 2.5 Flash | ~$0.30 / $2.50 | General workhorse tier |
| OpenAI nano tier | ~$0.10–$0.20 input | Cheapest branded-model volume play |
| Anthropic Sonnet-class | ~$3 / $15 | Mid-frontier reasoning; flat-rate long context |
List pricing varies and changes fast, so treat these as shapes, not quotes. Each vendor’s shape is different. Google’s Gemini line is the volume-pricing aggressor — Flash-Lite at roughly $0.10 in / $0.40 out is hard for any self-hosted small model to beat once you count engineering time. But note the direction is not uniformly down: Google raised Gemini 2.5 Flash list pricing during 2025 when it consolidated tiers, a useful reminder that a business case built on someone else’s price list carries repricing risk in both directions.
OpenAI competes hardest at the cheap end — its nano-tier models undercut nearly everything for simple high-volume tasks, plus roughly 90% off cached input reads and a 25% batch discount. Anthropic is rarely the cheapest per token; its case is quality per dollar at the mid-frontier tier and flat-rate pricing on very long context windows, which matters if your workload is document-heavy. Both Anthropic and OpenAI now discount cached prompt reads by about 90%, which quietly demolishes naive break-even math: if 70% of your tokens are a cached system prompt, your effective API rate is a fraction of list.
Takeaway: never compare self-hosting against API list prices. Compare it against your blended effective rate after caching and batching — that number is often 3–5x lower.
The middle path: managed open models
The binary framing — own GPUs or pay OpenAI — misses where a lot of enterprises actually land. AWS is the clearest example of the middle path: Bedrock gives you per-token access to Anthropic and open-weight models with enterprise controls and no infrastructure, SageMaker lets you serve open models on capacity you rent but do not rack, and EC2 P5-class instances with Capacity Blocks cover the full self-managed route. AWS also fields its own Inferentia silicon, which can undercut NVIDIA per-token for supported models — at the cost of a smaller software ecosystem and porting work.
The honest critique of the middle path is margin stacking: a managed open model on Bedrock costs more per token than the same model on hardware you operate well. You are paying AWS to hold the operational risk. For many teams in the 20M–100M gray zone, that is a rational trade. How the three hyperscaler platforms compare on exactly this is covered in our Bedrock vs. Vertex AI vs. Azure Foundry breakdown.
The costs nobody puts in the spreadsheet
A “free” open-weight model is the most expensive line item in some AI budgets I have reviewed. The recurring omissions:
- Engineering headcount. Serving infrastructure, quantization, eval pipelines, and on-call for a production inference stack realistically consumes 2–4 senior engineers — $500K+ per year fully loaded before you serve a single token.
- Model churn. Open-weight leaders change every few months. Every migration re-runs evals, red-teaming, and capacity planning.
- Failover capacity. APIs bundle redundancy into the price. Self-hosters buy it twice.
- Peak-to-average ratio. You provision for peak; you pay for average. Bursty traffic quietly halves your effective utilization — and doubles your real per-token cost.
What to do at your volume
- Under 20M tokens/month: stay on APIs. Spend the energy on prompt caching and model right-sizing instead — that is where your 40% savings actually is.
- 20M–100M/month: run a 90-day bake-off. Price your real blended API rate after caching, then quote dedicated rented capacity for an open model that passes your evals. Do not buy hardware yet.
- 100M+/month sustained: self-hosting almost always wins on unit cost — if you can keep utilization above roughly 60% and staff the platform team. Rent before you buy, and hold the depreciation risk consciously.
The sentence for the meeting: below 20 million tokens a month the API is cheaper than your own idle GPUs, above 100 million self-hosting wins, and everyone in between should be negotiating — with both sides.
Frequently asked questions
Is it cheaper to self-host an LLM than use an API?
Only above sustained volume — roughly 100M+ tokens per month with 60%+ GPU utilization. Below about 20M tokens per month, managed APIs are almost always cheaper once idle hardware and engineering time are counted.
How much does it cost to run a 70B model per million tokens?
As of mid-2026, roughly $1–$2.50 per million tokens on 8x H100-class capacity at healthy utilization, with FP8 quantization near the low end. Poor utilization can multiply that by 5–10x.
Do LLM API prices keep going down?
Mostly. Effective prices fell roughly 40–60% since mid-2025 through cheaper tiers, caching, and batch discounts — but not uniformly. Google raised Gemini 2.5 Flash list pricing in 2025, so build repricing risk into any multi-year case.
What are the hidden costs of self-hosting LLMs?
Engineering headcount ($500K+/year for a credible platform team), redundant failover capacity, model migration and re-evaluation every few months, and the gap between peak provisioning and average utilization.
Enterprise Techie publishes vendor-honest analysis like this daily — get the brief by email, free.