How Much Does It Actually Cost to Run a Local LLM? (Euros per Million Tokens, Measured)

, so it’s basically free.” I’ve said that, you’ve probably said that, and it’s the kind of claim that sounds true until someone measures it. So I measured it.

The box is “ardi” — one openSUSE machine with a single RTX 3090 (24 GB). On it I ran a controlled benchmark: three local models served by ollama, each driven under an identical fixed workload, while a dashboard priced every run by the real GPU energy it burned. Not a thermal-design-power estimate, not a back-of-envelope guess  — power-sampled from nvidia-smi every 10 seconds and integrated over each run’s exact start→end window, at my real day/night electricity tariff: 0.30 BGN day / 0.18 BGN night (Bulgarian lev  — my actual utility rate; converted to EUR throughout this piece at the fixed ECB peg, 1 BGN ≈ €0.5113).

The result is one number per model: euros per million output tokens. Five of eight models I measured came in cheaper than a hosted cloud API. Three didn’t — and it’s not the three you’d guess from parameter count alone.

The claim I wanted to test

The folk wisdom is that local inference is the cheap option  — you bought the GPU, the marginal token is free, the cloud is the thing with the meter running. Every part of that deserves a number, so here’s the one I went after: the marginal energy cost of generating a token locally, model by model, and how it stacks up against what a hosted “Flash”-class API charges for the same token.

To make that a measurement and not an argument, I needed three things held constant: the same workload for every model, the same hardware, and a meter I trusted reading the GPU during each run. The third one is the hard part at home — which is where the instrument comes in.

The method

Three models, picked to span the size range I actually keep on the box: gemma3:1b (tiny), gemma4:26b (Google’s Gemma 4–25.8B params under the hood; ollama just rounds it to “26b” in the tag), and gemma3:27b (the older big one). All three are Q4_K_M-quantized GGUF weights pulled through ollama, so the comparison is like-for-like on quantization. Each got the same fixed workload — a loop of 256-token generations cycling through five fixed prompts — sustained for ~4 minutes (240 s) so the GPU reached a steady state rather than being judged on a cold first call.

While each model ran, I tracked it as a priced experiment in HomeLab Monitor — my own dashboard, open source, MIT, one container. It samples GPU power from nvidia-smi every 10 s, and for a tracked run it integrates power over the run’s start→end window into kWh, then multiplies by whichever tariff was in effect (day vs night) — in the currency I configured it with, BGN. That converts “what did this model cost?” from a guess into a measured figure; I then convert to EUR for readability.

Then the arithmetic that turns a 4-minute run into a comparable rate:

€ per 1M output tokens = (run_cost_BGN × 0.5113) ÷ (output_tokens / 1,000,000)

The three-model generation benchmark below is reproducible end to end. Here’s how to stand it up on your own box.

Step 1 - bring up the monitor (one container)

On the machine with the GPU:

COMPOSE_URL=https://raw.githubusercontent.com/SikamikanikoBG/homelab-monitor/main/docker-compose.yml
curl -fsSLO "$COMPOSE_URL"
docker compose up -d

Open http://<hub>:9800. The host it runs on shows up immediately — CPU, RAM, disks, and (if it sees one) the GPU with per-model VRAM and live power draw. No config. This is the meter. Check Settings for the currency and tariff it’s configured with before you trust any € figure it shows you.

Step 2 - mint an ingest key

The benchmark script needs to push runs into the hub, so it needs a key. Settings → Integrations → Create key. The key (an hlm_… string) is shown once — copy it then, you won’t see it again. Keep it in an env var; don’t paste it into the script.

export HOMELAB_KEY="hlm_xxxxxxxxxxxxxxxxxxxx"

The same panel has a Download client link for the Python file you’ll use in Step 3, and a copy-paste quickstart — so this one screen covers the next two steps.

Step 3 - grab the tiny client

The hub serves its own Python client. It’s stdlib-only — nothing to pip install:

CLIENT_URL=http://<hub>:9800/static/homelab_run.py
curl -O "$CLIENT_URL"

That one file is the whole tracking API. The pattern is deliberately small: configure once, then wrap any block of work in a run context manager and log metrics inside it.

import homelab_run as homelab

homelab.configure(url="http://<hub>:9800", key="hlm_...")

with homelab.run("llm-cost-gemma3-1b",
                 params={"model": "gemma3:1b", "engine": "ollama",
                         "gpu": "RTX 3090 24GB", "num_predict": 256}) as r:
    # ... do the work ...
    r.log_metric("tokens_per_sec", tps, step=i)

When the with block opens, the hub marks a run start; when it closes, it marks the end and prices the GPU energy burned in between. The window is the measurement.

Step 4 - write the benchmark

The workload is just ollama’s /api/generate in a loop, reading the fields ollama hands back so we get real tokens-per-second rather than a wall-clock approximation. The two fields that matter are eval_count (tokens generated) and eval_duration (nanoseconds spent generating them):

import json, time, urllib.request

OLLAMA = "http://localhost:11434"

def ollama_generate(model, prompt):
    body = json.dumps({
        "model": model, "prompt": prompt, "stream": False,
        "options": {"num_predict": 256, "temperature": 0.7},
    }).encode()
    req = urllib.request.Request(OLLAMA + "/api/generate", data=body,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=600) as resp:
        return json.loads(resp.read())

Then drive each model under load for a fixed duration, wrapped in a tracked run:

def bench(model, duration_s=240):
    ollama_generate(model, "Say hello.")  # warm up - exclude the cold start
    run = homelab.run(f"llm-cost-{model.replace(':', '-')}",
                      params={"model": model, "engine": "ollama",
                              "gpu": "RTX 3090 24GB", "num_predict": 256,
                              "workload": "fixed-prompt-loop"},
                      tags=["llm-cost", "benchmark"])
    tot_out = 0
    t0 = time.time(); step = 0
    with run as r:
        while time.time() - t0 < duration_s:
            res = ollama_generate(model, PROMPTS[step % len(PROMPTS)])
            ec, ed = res.get("eval_count", 0), res.get("eval_duration", 1)  # ed in ns
            tps = ec / (ed / 1e9) if ed else 0.0
            tot_out += ec
            r.log_metric("tokens_per_sec", round(tps, 2), step=step)
            step += 1
        r.log_params({"total_output_tokens": tot_out})
    return run.id, tot_out

A warm-up call before the timed window keeps the cold-start latency out of the measurement. The time.sleep(15) I put between models (not shown) keeps each priced window cleanly separated so one model’s tail doesn’t leak into the next one’s bill.

Step 5 - read the run back, priced

Run the script for all three models and open the Experiments tab. Each run is now a row carrying its real energy and cost:

Click into a run and you get the GPU power curve for that exact window, with the priced caption underneath — the raw material the euro figure is computed from:

The hub exposes each run’s cost programmatically too, so the final division can happen in the script:

pulled = homelab.pull(run_id)  # cost, energy_kwh, avg_w, power curve - cost is in BGN
eur_per_mtok = round((pulled["cost"] * 0.5113) / (tot_out / 1e6), 4)

The numbers

I later went back and added five more models - GLM-4.5-Air (106B), DeepSeek-R1-Distill (32.8B), Seed-OSS (36B), Devstral (24B), and Qwen3-Coder (30.5B) - pulled from a separate coding-agent benchmark I’d already run on the same box (same GPU, same tariff, same cost÷tokens method, but real multi-step agent tasks instead of a dedicated 240-second generation loop — noted where it matters, in the caveats below). Eight models, one scale:

And the cloud reference I was measuring against (list prices, June 2026): a hosted Flash-class model - Gemini 2.5 Flash, GPT-4o-mini - runs about $0.60 per 1M output tokens (~€0.55); the cheapest tier going (Gemini 3.1 Flash-Lite) is around $0.40.

The surprise

Five of the eight are cheaper than the cloud reference. Three are not — and the three that aren’t are not the three you’d pick by parameter count.

GLM-4.5-Air is the biggest model on the list, 106B parameters, and it costs €1.040/1M — genuinely more than the cloud API, electricity alone, before a cent of GPU purchase price is counted. That part matches the original intuition: big model, slow, expensive.

But DeepSeek-R1-Distill is barely a third of GLM’s size — 32.8B parameters — and it’s the most expensive model on the entire list, at €1.526/1M. It draws less power than gemma3:27b (155 W vs 283 W) — and is still the most expensive per token of any model tested.

One methodology note that matters here, upfront rather than buried: the three gemma models were measured on a dedicated 240-second sustained-generation benchmark I built for this piece. The other five — including DeepSeek-R1-Distill — come from an earlier, separate benchmark of real multi-step coding-agent tasks (tool calls, multi-turn reasoning) on the same GPU and tariff, using the identical cost÷output-tokens formula. That distinction is the whole reason DeepSeek looks the way it does: its raw generation speed when actively producing tokens is a respectable 6.7 tokens/sec. But raw generation speed isn’t what the meter charges for. The meter charges for wall-clock time on the card, and DeepSeek — a reasoning-distilled model - spends most of that time deliberating between generations, not generating. Its effective delivery rate (output tokens ÷ total run time, deliberation included) works out to 3.7 tokens/sec, the lowest of any model tested — well below Seed-OSS’s effective 7.1 and GLM’s 5.0. That’s what the price actually tracks.

Size was never the mechanism. Effective speed was — and effective speed is not the number these models are usually benchmarked at.

The mechanism

Each token costs watts ÷ throughput. That’s the whole equation — the trick is using the right throughput number:

gemma3:1b is fast (136 tok/s) and light (154 W) — cheap on both axes, cheapest overall. Devstral pulls the most power of any model tested (320 W) but stays mid-table because it’s still reasonably fast. For the three purpose-built gemma models, raw generation speed is the number that sets the price - watts ÷ raw tok/s reconstructs the cost ratio almost exactly. For the five agentic-benchmark models, the number that sets the price is effective wall-clock throughput, not the raw generation speed they’d normally be quoted at — and by that number, Seed-OSS (7.1 tok/s effective), GLM-4.5-Air (5.0), and DeepSeek-R1-Distill (3.7) are the three slowest models tested, in exactly the order their cost ranks them. None of them have unusual power draws (141–186 W, actually lower than several cheaper models); they’re just slow once tool calls and reasoning gaps are counted, and a model that’s slow pays for every second it holds the card, regardless of parameter count.

What I am *not* claiming

This is the part that keeps it honest, so I’ll be blunt about the edges:

· This is GPU electricity only - measured, marginal, and tiny. A full run rounds to a fraction of a cent. I’m not measuring CPU or DRAM (RAPL was unreadable on this host, so those came back null), and I’m explicitly not counting the GPU’s purchase price, idle draw, cooling, or my time. The €/1M figures are the marginal energy rate, useful for “which model is cheaper to keep generating,” not a total cost of ownership.

· The real cost of local is the GPU you bought, not the watts. A 3090 amortizes only under high, steady utilisation. A box that’s busy a few minutes a day is mostly paying for idle silicon, and no electricity figure captures that. The marginal-energy lens is honest about its own scope: it’s the floor, not the bill.

· Two measurement methods (see “the surprise” above for why it matters). The three gemma models: a dedicated 240-second sustained-generation benchmark. The other five: a separate real coding-agent benchmark, same cost÷tokens formula, but effective wall-clock throughput rather than raw generation speed — which is arguably more representative of how these models actually get used, not less, since real agent workloads include tool calls and deliberation a pure generation loop never would.

· One GPU, one engine, quantized GGUF weights. Your tariff, model, batch size, quantization and card will move every number here. The table is my box; the method is the transferable part.

So treat the absolute euros as illustrative and the shape as the finding: cost per token tracks effective delivery speed, and that doesn’t track parameter count the way you’d expect.

The takeaway

Pick the smallest, fastest model that clears your quality bar — that’s where “local” actually pays, by a wide margin, and it has nothing to do with picking a small model because it’s small. Qwen3-Coder at 30.5B is cheaper per token than gemma4:26b, and gemma3:27b at 27B beats every reasoning-style model on the list despite being a plain dense model with no special efficiency tricks. The variable that matters is tokens delivered per wall-clock second, full stop — not the raw generation-speed number a model is usually benchmarked at — and the only way to know it for your model and your box is to measure it.

The instrument I used is HomeLab Monitor — open source, MIT, one container — and the run-pricing above is exactly what its Experiments tab does out of the box:

Add your GPU host, mint a key, and your next benchmark prices itself — in whatever currency you actually configured, which is worth checking before you publish the number.

So: when you reach for the local model over the API, do you actually know your per-token number, in the right currency — or are you, like I was, just assuming the GPU makes it free?