Hosted API, your own GPUs or both? Five questions that decide where a language model should run, and what each option really asks of you.
The first AI project in an organisation almost always runs on a hosted API, and that is the right call. It costs little to start and it answers the only question that matters at that stage: is this useful?
The harder question arrives later. Usage grows and the monthly bill with it. Legal asks where the prompts go. A customer in healthcare, government or defence says their data may not leave the building. At that point somebody proposes buying GPUs, and the discussion turns into a matter of belief.
It does not have to be. Five questions settle most of it.
The three options
- A hosted API. A provider runs the model. You pay per token and manage nothing.
- Your own infrastructure. You run open-weight models on GPU servers you control, in your own data centre or rack, up to a fully air-gapped system with no network path out.
- Hybrid. Some workloads run on your hardware and others go to a hosted API.
Question 1: what goes into the prompt?
Start with the data, because it can end the discussion on its own. Public documentation and marketing copy can go anywhere. Patient records, personal data, source code, defence material and anything under a strict contract cannot.
Choosing a European cloud region helps, but it is only part of the answer. Three things determine whether a deployment is sovereign in a way you can prove:

- Model provenance: is the model served within EU jurisdiction, or reached through an API from a non-EU company?
- Hardware sourcing: where does the equipment come from, and who can manage it remotely?
- Operational access: who holds admin credentials, support tunnels or telemetry access?
We went into this in what EU AI sovereignty actually requires. If your data falls into the strictest category, the answer is on-premise, and the remaining questions are about how.
Question 2: how much, and how steadily?
A hosted API has no fixed cost. You pay for what you use, and nothing when you use nothing. That makes it the cheapest option at low or unpredictable volume.
Your own hardware is the reverse: a fixed cost whether you use it or not. It pays off when the GPUs are busy. Many organisations already own powerful GPU servers and run them well below capacity, which is the most expensive way to do inference.
So estimate two numbers before comparing prices: how many tokens you expect per day in a year's time, and how evenly that load is spread. A steady load around the clock favours your own hardware. A spike every Monday morning does not.
Compare like with like, too. The relevant figure is cost per token delivered, not the price of a server. We explained why in from electron to inference.
Question 3: how fast must it answer?
For a nightly batch job, latency is irrelevant. For an assistant that speaks, a digital human, a robot or a vision system on a production line, every millisecond counts.
Running the model yourself removes the rate limits and the shared-tenancy latency of a public service, and lets you tune the stack for your own workload. Where the answer has to be produced on site with no round trip at all, the model has to run at the edge, on the device.
Question 4: which model do you actually need?
Open-weight models such as Mistral and Llama-based models handle a large share of everyday tasks well: summarising, extracting, classifying, answering questions over your own documents. Served on a modern inference stack, their latency and throughput are now competitive with hosted frontier APIs for most real-world use cases.
The largest proprietary models are a different matter. They are only available from their providers. If your use case really needs one of them, that part of your workload stays hosted.
Do not decide this by reputation. Take fifty real examples of your task and run them through an open model. The result is usually clearer than any benchmark.
Question 5: who will run it?
This is the question that is skipped most often. Your own infrastructure means your own operations:
- sourcing and installing hardware
- choosing and tuning a serving runtime, such as vLLM, TensorRT-LLM, NVIDIA NIM or Triton
- orchestration, usually on Kubernetes
- monitoring GPU utilisation, memory, temperature and power draw
- updates, security and access management

The serving stack matters more than most teams expect. The same hardware delivers very different throughput depending on the runtime and how it is configured. If you have no team for this, either work with a partner or stay hosted until you do.
A decision table
| Your situation | Lean towards |
|---|---|
| Low or unpredictable volume, non-sensitive data | Hosted API |
| Sensitive data, steady and high volume | Your own infrastructure |
| Data that may never touch a network | Air-gapped, on-premise |
| A mix of sensitive and ordinary workloads | Hybrid |
| A task that needs the largest proprietary models | Hosted API, with control over what is sent |
| Real-time responses on site | Edge, close to the device |
What hybrid looks like in practice
Hybrid is a common outcome, and it works when the split follows the data. Workloads that touch sensitive data run on your own hardware. Everything else goes to whichever option is cheapest. The application does not need to know the difference if you put a routing layer in between.
It is also the lowest-risk way to move. Prove the use case on a hosted API, measure the real volume, then move the workload that justifies it.
Four common mistakes
- Buying GPUs before knowing the workload. The hardware should follow the measurements.
- Treating an EU region as sovereignty. It is a start, not the whole answer.
- Comparing an API price with a hardware price. Add the people, the power and the idle hours.
- Ignoring the serving stack. A badly configured runtime wastes most of what the hardware can do.
Where regulation comes in
GDPR, NIS2, DORA for financial services and the sector rules in healthcare and defence rarely say "on-premise". What they ask is that you know where data goes, control who can reach it and can show both. Sometimes a hosted service with the right contract meets that. Sometimes only your own infrastructure does. That is a question for your legal team, and the answer is one of the inputs to question 1.
A place to start
If you already run models on your own hardware, or are about to, we offer an assessment that returns a GPU utilisation and cost analysis, latency and throughput benchmarks, a recommended inference engine per workload and a production roadmap. See what it covers on our AI infrastructure page.










