The Private Generator: Why Local LLMs Aren't for Everyone
I have a small homelab. It consists of four mini-PCs humming quietly in a neat little cluster. I love them. They handle my daily services, my tests, and my automations without raising my electricity bill or heating up the room. But every time a new open-source model drops on Hugging Face, I watch people online tell everyone to “throw away the cloud” and run it themselves on their own metal.
Nice picture. Wrong picture.
The idea of running your own model, completely offline and independent of any corporate API, is appealing. You own the machine, choose the model, and decide where the data goes. But once you move beyond tinkering, the hardware starts setting the rules.
Running a local model is like running a private fuel-powered generator in your back garden to power your house. It takes money up front, ongoing maintenance, fuel, and manual tuning. If the generator breaks, you are the mechanic. For plenty of workloads, plugging into the national grid is the more practical choice.
The interesting part is the machinery underneath that choice.
The fuel tank is measured in VRAM (and HBM)
Before a model can produce an answer, its weights need to be in memory that the inference engine can reach. Generation then happens one token at a time. A token may be a whole word, part of one, punctuation, or even a byte; the Hugging Face tokenizer guide shows why “one token” and “one word” are not interchangeable.
For a dense model serving one request, each decoding step moves the model weights and the growing key-value cache through the system. That often makes generation memory-bandwidth-bound. The prompt-processing phase, larger batches, and mixture-of-experts models behave differently, so bandwidth is not the only limit. NVIDIA’s inference guide separates those phases and accounts for both weights and the key-value cache.
Consumer graphics cards use fast graphics memory such as GDDR6X or GDDR7. The RTX 5090, for example, has 32 gigabytes of GDDR7 with 1,792 GB/s of memory bandwidth. Data-centre accelerators use High Bandwidth Memory (HBM) and high-speed GPU interconnects to push capacity and bandwidth much further. System RAM and unified-memory machines can run local models too, often with more capacity per euro, but usually at lower generation speed than a well-fed GPU. This is the generator’s fuel line: capacity gets the model into the machine; bandwidth helps determine how quickly it can produce the next token.
The weights of a dense 70-billion-parameter model (70B) take about 140 gigabytes at 16-bit precision. The runtime also needs memory for the key-value cache and other working data, and that cache grows with context length and concurrent requests. NVIDIA uses the same 140-gigabyte estimate for Llama 3 70B. A high-end consumer card like the RTX 5090 has 32 gigabytes.
Raw capacity alone would take five cards, before the runtime overhead. That is not a sensible RTX 5090 design: the card has no NVLink, so a multi-GPU inference engine must also work around the slower PCIe link between cards.
One way around the capacity problem is quantisation. Think of it as storing a less precise measurement for each part of the fuel mixture. Moving weights from 16-bit to 4-bit cuts the raw weight storage to about 35 gigabytes; scales, metadata, the key-value cache, and runtime buffers push the real requirement higher. A 70B-class model can therefore fit across two 24- or 32-gigabyte cards, although fitting is not the same as running it quickly.
Lower precision can reduce quality, but it does not impose one fixed “intelligence tax.” The result depends on the model, quantisation method, and task. Some modern 4-bit methods report near-FP16 accuracy on their test suites; naive conversions can lose much more. MosaicQuant’s results are a useful example of that range. Test the exact model on the work you care about.
You can run smaller models, of course. An 8-billion-parameter model (8B) fits on a single consumer GPU, and tools like llama.cpp make it straightforward to try. Small models can be excellent at narrow, well-tested jobs. They are not automatic substitutes for a hosted general-purpose model, but the reverse is also true: paying for a large model is wasteful when a smaller one already passes your evaluation.
The true cost of off-grid living
Let’s talk about money. If you are a company, your accountant cares about the difference between capital expenditure (CapEx) and operational expenditure (OpEx).
Cloud services are operational expenditure, but the bill depends on how you buy them. An API charges for the tokens you use. A team subscription charges for the seats whether they are busy or idle. Both avoid the up-front hardware purchase, but neither is free of administration or the risk of extra usage charges.
A local GPU rig is capital expenditure. You buy the hardware up front. At the time of writing, individual RTX 5090 cards are listed at roughly 4,000 euros in Germany, while a ready-built dual-RTX-5090 workstation is listed at 17,205 euros including German VAT. The old 5,000-to-7,000-euro estimate no longer buys this class of machine.
And that hardware does not sit quietly.
Power comes first. The cost model below assumes 100 watts at idle. Under load, the two GPUs alone can draw up to 1,150 watts before the CPU, memory, fans, and power-supply losses are counted; NVIDIA rates each RTX 5090 at 575 watts.
Then there is heat. Removing it takes more electricity, and the machines still need rack or office space, power distribution, and network ports. Their resale value is uncertain too, so the comparison below assumes rather than promises a future value.
Finally, someone has to install the machines, deploy the inference stack, patch it, monitor it, replace failed parts, and recover it when an update goes wrong. That time belongs in the cost model.
My mini-PC cluster is busy throughout the day with CI/CD jobs, network services, and automations. It earns its shelf space.
But a dedicated GPU rig for local AI is a different beast. Unless you are running inference requests constantly, that expensive hardware sits idle most of the time, silently drawing standby power, depreciating in the corner, and waiting for the next prompt.
Buying it is not the same as buying it cheaply
Supply is tight, but “you cannot buy a GPU” is too simple. NVIDIA says memory availability and other supply constraints may limit production, and that gaming supply constraints will remain a headwind beyond the first quarter of its 2027 financial year. That is the manufacturer’s risk statement, not a promise that every card is unavailable. NVIDIA’s annual filing spells out the constraint.
German retailers still list RTX 5090 cards and complete dual-GPU workstations. The practical problem is price and lead time: the machine may exist, but not at the launch price or on the date your project needs it. Treat availability as a live procurement input, not a permanent fact about the market.
The numbers on the desk
For a cost comparison, every number needs a boundary and an assumption. This scenario covers five developers over five years. The local side uses two dual-RTX-5090 workstations, each hosting one dense 70B-class model quantised to 4-bit across its two GPUs. That gives the team two model replicas. Five people must share them through batching or a queue, and the actual speed depends on the model, context, inference engine, and PCIe overhead.
The cloud side uses five Claude Team Premium seats, billed annually. Those seats provide an interactive product with current hosted models, including Claude Sonnet 5; they are not private API capacity and do not guarantee the same latency or output as the local service. This is a bounded cost scenario, not a claim that the two systems do identical work. A production API workload needs a separate token-volume comparison.
Prices exclude recoverable VAT, and dollars are converted at the ECB reference rate of 1 euro to 1.1430 dollars on 10 July 2026. The local hardware line takes two workstations at 17,205 euros each, removes 19 percent German VAT, and rounds the result: 28,900 euros.
The local power estimate assumes 100 watts per rig while idle, 1.25 kilowatts per rig under load, and ten percent of the five years at that load. It uses the latest published German non-household average of 19.22 cents per kilowatt-hour, plus twenty percent for cooling and power-distribution overhead. Labour costs 150 dollars an hour: 40 hours to build and deploy the local platform, then two hours a month to patch, monitor, test, and repair it. The cloud needs 16 hours to configure the workspace and 30 hours of administration across the five years.
The remaining assumptions are deliberately plain: 50 euros a month for the local machines’ share of space, power distribution, and networking; ten percent of the hardware price for repairs and spares; six percent a year for tied-up capital; and ten percent resale value after five years. The cloud side includes the smaller capital cost of paying annually in advance and a ten-percent reserve for usage beyond the seats’ included limits. Normal office internet is shared by both options, so it is not an incremental cost on either side.
| Incremental cost over five years, excluding VAT | Two local GPU rigs | Claude Team Premium |
|---|---|---|
| Hardware purchase | 28,900 € | 0 € |
| Initial setup (40 h local / 16 h cloud at $150) | 5,250 € | 2,100 € |
| Ongoing operations (120 h local / 30 h cloud at $150) | 15,750 € | 3,950 € |
| Electricity | 3,620 € | 0 € |
| Cooling and power overhead | 720 € | 0 € |
| Space, power distribution, and network allocation | 3,000 € | 0 € |
| Repairs and spare-parts reserve (10% of hardware) | 2,890 € | 0 € |
| Software licences (vLLM/Ollama stack) | 0 € | 0 € |
| Claude Team Premium subscription | 0 € | 26,250 € |
| Additional-usage reserve (10% of subscription) | 0 € | 2,625 € |
| Cost of tied-up capital (6%) | 4,335 € | 785 € |
| Estimated hardware resale value after five years | -2,890 € | 0 € |
| Five-year TCO | 61,600 € | 35,700 € |
Depreciation does not appear as another full hardware charge. The cash left the company when it bought the machines, so adding the complete purchase price and complete depreciation would count the same loss twice. The table instead credits a conservative ten-percent resale value at the end.
Claude Team Premium currently costs 100 dollars per person per month when billed annually. That is 30,000 dollars for five people over five years before tax and extra usage. It is not unlimited: Anthropic applies session and weekly limits, then sells additional usage credits when a team needs more. The ten-percent reserve makes that risk visible without pretending we know the team’s future token consumption.
Under these assumptions, the local option comes to about 61,600 euros against 35,700 euros for Claude Team Premium. Change the workload, utilisation, labour rate, model, or privacy requirement and the answer can move. Queueing time, model quality, cloud outages, and the value of keeping data on-site may matter more than the hardware bill, but they need measurements from the actual workload. Inventing them would make the table look complete while making it less honest.
Who keeps the generator running?
When you plug into an API like OpenRouter or Anthropic, you are paying someone else to handle the engineering. They manage the server clustering, the load balancing, the hardware failures, and the software updates.
When you go local, you are the operations team.
You have to set up and maintain the inference engine (like vLLM or Ollama). You have to manage model weights, monitor memory consumption, configure context caching to keep things fast, and handle concurrent requests when multiple users hit your local server at the same time.
If a server crashes at 3:00 AM because of a memory leak or a corrupted weight file, your system is down until you wake up and fix it. That labour is not a footnote. In the five-year comparison above, setup and operations add roughly 21,000 euros to the local side before a single hour of developer waiting time is counted.
The power grid works for a reason
I am not saying local models are useless. They make sense for offline work, measured workloads that run often enough to pay back the hardware, or projects where data must stay on-site. They are also great for tinkerers, operators, and people who, like me, enjoy watching those tiny LED lights blink on their shelf.
But for many companies and individuals, local models are an expensive answer to a problem they have not measured yet.
The power grid exists because centralising massive generators is incredibly efficient. It allows providers to buy hardware in bulk, manage cooling at scale, and distribute the cost of maintenance across millions of users. You pay a metered API bill or a fixed seat price instead of buying the generator.
There’s no magic to running models. They run on compute, memory, and electricity. Before you buy a pallet of GPUs, ask yourself: do you want to build a utility company, or turn on the lights?
The practical model
Let’s compress this down into a simple map:
- Cloud services are the national grid: An API meters usage; a team plan rents seats. Either way, the provider owns and operates the generator.
- VRAM and HBM are the fuel tank and fuel lines: Fitting the model’s weights is only half the battle. Memory bandwidth, the key-value cache, context length, and concurrent requests help determine how fast the system can answer.
- Quantisation changes the mixture: It stores weights with fewer bits so the model fits on smaller hardware. The quality trade-off depends on how it is done and what you ask the model to do.
- Local hosting is self-reliance: It keeps the machinery and data under your control, but you pay for hardware, power, cooling, space, repairs, capital, and the engineering time required to keep it alive.
The quiet on the shelf
My mini-PC cluster is still humming. I like knowing those nodes are there, independent and completely under my control. But when I need to get real work done, write code, or run complex analysis, I don’t use them. I plug into the grid.
I don’t want to spend my hours cleaning the generator. I want to write.