I run a service that routes AI inference to the cheapest available GPU. So I spend an unreasonable amount of time staring at provider pricing dashboards. And the thing I keep learning is that the sticker price on an AI model tells you almost nothing about what inference actually costs you.
Take a 70B parameter model. Provider A charges $0.60 per million input tokens. Provider B charges $0.90. Provider A is cheaper, right?
Not necessarily. Provider A has cold starts that add 3-8 seconds to the first request. Provider B keeps models warm. Provider A queues requests during peak hours - your p95 latency spikes to 30 seconds. Provider B does not. If you are running a chatbot, those cold starts translate directly into users closing the tab.
Spot prices move like electricity
GPU spot prices change by the hour. An H100 costs $2.10/hr on one provider right now. Same GPU, same hour, $3.80/hr on another. Tomorrow those numbers will flip. If you are locked to one provider, you are overpaying roughly half the time.
I did not know this until I started tracking it. I had picked a provider, set it up, and forgotten about it. For about a month. Then I looked at the bill and started understanding why it was higher than I expected.
The price per token is just one number. Cold start penalties, failed request costs, output-vs-input token ratios, and spot pricing swings all compound. A 2% failure rate at scale adds 3-5% to your effective cost because each failed request burns compute before failing, and then you retry.
What actually cut the bill
I tried a spreadsheet first. Listed the providers, compared prices, switched to the cheapest one. The bill went down. For about a week. Then the prices moved and I was overpaying again.
So I built a router. Change one line in your code - the API base URL - and it sends each request to whichever provider is cheapest right now and actually responding. That became Smart Inference.
Beyond the routing, the things that made the biggest difference:
Use the right model for the job. A 7B model handles classification and simple Q&A just as well as a 70B. We use Haiku for routine work and only reach for the larger models when judgment matters. That alone cut costs more than any provider switch.
Batch when you can. Background processing, nightly reports, content generation - none of these need real-time responses. Batch jobs fill GPU idle time and cost 2-3x less.
Cache. Identical prompts with identical inputs happen more often than you think. A Redis cache in front of the inference endpoint cut 15% of our requests.
Constrain output. System prompts that ask for JSON instead of prose pay for themselves immediately. Output tokens cost 3-5x more than input tokens on most providers, and verbose model responses add up fast.
Where the market is heading
AI inference in 2026 looks a lot like cloud computing in 2010. Prices are dropping fast, roughly 10x cheaper than two years ago for the same quality. But the market is fragmented and nobody has won yet. The winning strategy is the same one that worked for cloud: do not lock in, route dynamically, measure everything.
And if you do not want to build all of that yourself - well, that is what we built Smart Inference for. But honestly, even if you build your own, the important thing is to stop treating inference as a fixed cost. It is not.