Five Things I Learned Renting a GPU Instead of Calling an API
I rented a GPU and ran an open-weight model on it myself instead of hitting a hosted API. It was cheaper than I expected and slower to set up than I expected — here's the honest list.
I've been calling hosted LLM APIs for most of my work, same as everyone. Recently I set one up the other way: rented a bare GPU instance and ran an open-weight model on it myself, behind an OpenAI-compatible endpoint I control end to end. Not a laptop with a small quantized model — an actual full-size model on a rented data-center GPU.
I did it mostly out of curiosity about the tradeoff, and it turned into a small pile of lessons I hadn't seen written down anywhere.
1. The address changes under you
Rented GPU instances often have their inference endpoint tied to the instance's own ID. Redeploy the instance, get a new ID, get a new URL. Nothing tells you this happened — your app just quietly starts talking to a server that no longer exists until something downstream fails and you go looking for why.
2. Restart isn't a promise
I hit a case where a stopped instance couldn't restart because the underlying host machine didn't have a free GPU anymore. No persistent storage attached meant the fix was faster than debugging: throw it away, spin up a fresh one from a saved template.
3. You inherit the host's problems, not just its GPU
On the cheaper tier of the provider I used, some hosts run an older CUDA driver that crashes the inference server outright at boot. There's no way to request a specific driver version up front — you create the instance, check what you got, and try again if it's wrong. Paying more for the "reliable" tier of the exact same provider made this problem disappear entirely, which was a useful data point in itself: the premium tier isn't selling you speed, it's selling you not having to care about this.
4. Cold start is a real number, not an afterthought
Weights downloading, the inference engine compiling itself, the server actually coming up — about five minutes, start to finish, in my setup. A hosted API trains you to never think about this. Running your own model means it's now part of your deploy loop, and anything that depends on the model needs to survive that gap.
5. Context length and concurrency are now your problem
Want a longer context window? That's memory the GPU can't spend on serving more requests at once. A hosted provider absorbs that tradeoff for you, invisibly, in their pricing. Self-hosting means you're the one deciding, explicitly, which one you'd rather have less of.
Was it worth it
In pure dollar terms, yes — a rented GPU is genuinely cheaper than API tokens once you're doing any real volume. In time terms, it cost me a weekend I'd budgeted for something else, entirely on infrastructure quirks that a hosted API would have made invisible. I'd do it again, knowing what I know now it'd take a fraction of the time — which is basically what this list is for.
More on what I'm actually building on top of it, later.
