I accidentally stumbled on spot market pricing for AI compute. So naturally, I built a service around it.
The short version: GPU prices vary wildly between providers. Same model, same output, different costs depending on when and where you run it. An H100 on one provider costs $2.10/hr right now. Same GPU on another, $3.80/hr. Tomorrow those numbers will be different again.
I found out the hard way. I was running AIVory Guard on a single provider, picked because it was cheapest when I checked. A month later the bill was 40% higher than I expected. The prices had moved and I had not noticed.
So I opened a spreadsheet. Listed every provider, their prices, their latency, their uptime. Switched to the new cheapest one. The bill went down. For about a week.
That spreadsheet was accurate on Tuesday and wrong by Friday. Spot GPU pricing moves like electricity markets. Providers have idle capacity between reservations and sell it at a discount, sometimes 60% below on-demand. But the price floats with demand, and not all providers spike at the same time.
From spreadsheet to router
I wrote a script that checked prices hourly and picked the cheapest. Then I added health checks because the cheapest provider was also the one that went down most often. Then I added latency filters because some providers are fast but far away. Then I added automatic failover because when a provider fails mid-request, you need to retry somewhere else, not wait.
By that point it was not a script anymore. It was a service. Change one line in your code - the API base URL - and the router sends each request to whichever provider is cheapest right now, has capacity, and meets your latency requirements. We call it Smart Inference.
The routing decision takes about 50ms. That is the overhead. For real-time chat, it is noticeable. For batch processing, content generation, background jobs, it is nothing.
The hard part was not routing
The routing logic was straightforward. The hard part was health monitoring.
GPU providers go down. They hit capacity limits. They have maintenance windows that they announce on Twitter instead of through their API. One provider returned 200 OK with empty responses for six hours before we noticed. Another started queueing requests silently - the response would come, eventually, after a 45-second wait.
So we built health checks that send real inference requests every 60 seconds. Small ones, a few tokens each. We measure actual response time, actual output quality, actual error rates. A provider that slows down gets deprioritized before it fails completely.
The other hard part: pricing data. Not every provider publishes real-time spot prices through an API. Some update their pricing page once a day. Some negotiate custom rates. We ended up building scrapers, negotiating API access, and in some cases just monitoring our own billing data to infer current rates.
Where it does not work
Fine-tuned models are stuck on one provider. If you fine-tuned on Lambda, you cannot route to CoreWeave - they do not have your weights. We can route between providers hosting the same base model, not custom weights.
Latency-critical endpoints with sub-100ms requirements need a dedicated instance close to the users. The 50ms routing overhead plus potential geographic routing makes this a bad fit.
Provider-specific features - structured output guarantees, extended context windows, specific function calling implementations - break if you route away from the provider that offers them.
The numbers
For mixed workloads, average savings are 30-40% compared to a single provider. For batch-heavy workloads with flexible latency, we have seen up to 55%.
The service runs on prepaid credits. No subscription, no minimum. You deposit credits and we deduct at the actual spot rate plus a small routing fee. If you want to try it - aivory.net/smart-inference. Change one line. See what the bill looks like after a week.
Anyway, I will keep you posted as always ;)