Self-hosted AI for European SMBs: what actually works in 2026, cost and GDPR

Every few weeks a European company asks us some version of the same question: "Can we run our own AI so our data never leaves our control?" The honest answer is yes, and often you do not need to. Self-hosting large language models is now genuinely practical for small and mid-sized businesses, but it is a trade-off between control, quality, cost and operational effort. This article is the framework we use when advising clients in 2026.

We run client workloads on our own dedicated hardware in German data centers, and we operate Namiru, our EU-hosted AI customer support product for European SMBs, so we see both sides: the appeal of full control and the cost of actually running it.

Three options, not two

The debate is usually framed as "OpenAI versus self-hosted". In practice there are three tiers:

For a large share of SMB use cases, an EU-hosted API with a solid data processing agreement, no training on your data, and deliberate data minimisation is enough. Self-hosting starts to pay off when at least one of these is true:

• A client contract or regulator prohibits sending data to third-party AI processors.
• The data is highly sensitive (health, legal, financial, HR) and you want to minimise the number of processors.
• You have steady, predictable volume, for example batch processing thousands of documents daily.
• You need a fixed model version that will not change under you.
• You want to fine-tune or tightly control the model's behaviour.

Choosing an open-weight model

The open-weight ecosystem is mature. The Llama, Mistral, Qwen and Gemma families all publish models from small (a few billion parameters) to very large. For SMB workloads, the sweet spot is usually a small or mid-sized instruction-tuned model that fits on one or two GPUs.

What we have found holds true:

• Task-specific quality beats leaderboard rank. Build an evaluation set of 50 to 200 real examples from your own documents and score candidates on it.
• Smaller models do well on narrow tasks. Extraction, classification, summarisation and retrieval-augmented answers rarely need a frontier model.
• Multilingual matters in Europe. Test in the languages your customers actually write in, including Slovak, Czech, Hungarian or Polish if relevant. Quality varies a lot between models.
• Read the license. "Open weights" does not always mean unrestricted commercial use.

Quantization trade-offs

Quantization stores weights with fewer bits, which cuts memory and often increases speed. As a rough rule, a model needs about 2 bytes per parameter at 16-bit precision, about 1 byte at 8-bit and about half a byte at 4-bit, plus headroom for the KV cache that grows with context length and concurrent users.

Always re-run your evaluation set after quantizing. Degradation shows up first on multi-step reasoning and less common languages.

Serving stack and GPU sizing

Inference servers

• vLLM: our default for GPU serving. Continuous batching, paged attention and an OpenAI-compatible API, so application code does not care whether it talks to a hosted API or your server.
• TGI (Text Generation Inference): Hugging Face's production server, a solid alternative with similar capabilities.
• llama.cpp: efficient C++ runtime for GGUF quantized models, runs on CPUs and modest GPUs. Great for low-volume internal tools.
• Ollama: developer-friendly wrapper for running models locally. Excellent for experimentation, less suited to multi-user production serving.

A minimal vLLM deployment looks like this:

# Sketch: serve an open-weight model with an OpenAI-compatible API
docker run --gpus all -p 8000:8000 \
  -v /srv/models:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model <org>/<instruct-model> \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

Put it behind your own gateway with authentication, rate limits and request logging that you control. Never expose the inference port directly to the internet.

Sizing in general terms

• A small model (up to roughly 8B parameters) fits comfortably on a single GPU with 24 GB of memory, even at 16-bit.
• A mid-sized model (roughly 20B to 35B) typically needs a 48 GB to 80 GB GPU, or 4-bit quantization on smaller cards.
• Large models (70B and above) generally need multiple high-memory GPUs, or aggressive quantization with slower throughput.

Concurrency and context length drive KV cache memory, so a model that "fits" for one user may not fit for twenty. Load test with realistic prompts before you commit to hardware.

RAG: where most of the value actually comes from

For company knowledge, retrieval-augmented generation usually matters more than the model choice. The model answers from documents you retrieve at query time, so it can cite sources, respect permissions and stay current without retraining.

A pragmatic stack:

1. Ingest: parse documents, split them into chunks that follow headings, attach metadata (source, date, department, access group).
2. Embed: use an open embedding model that you also self-host, so document text never leaves your infrastructure.
3. Store: pgvector if you already run PostgreSQL, Qdrant if you want a dedicated vector database with rich filtering.
4. Retrieve: hybrid search (keyword plus vector), filtered by the user's permissions, then rerank.
5. Generate: pass the top chunks with clear instructions to answer only from the sources and cite them.

# Sketch: permission-filtered retrieval with pgvector
rows = db.execute(
    """
    SELECT id, content, source
    FROM chunks
    WHERE access_group = ANY(%s)
    ORDER BY embedding <=> %s
    LIMIT 8
    """,
    (user.groups, query_embedding),
).fetchall()

The permission filter in that query is not optional. A RAG system that ignores document permissions is a data leak with a chat interface. When the data lives in live business systems rather than documents, an MCP server is often the better integration layer; see how to build a custom MCP server for company data.

GDPR and the EU AI Act: the practical checklist

We are engineers, not lawyers, so treat this as a starting checklist to review with your legal counsel.

• Data processing agreements with every processor: your hosting provider, any API provider, any monitoring service that sees prompts.
• Data residency: know where inference, logs, backups and vector indexes physically live. Self-hosting in the EU makes this easy to answer.
• Logging with intent: prompts and outputs often contain personal data. Decide what you log, redact where possible, restrict access and set retention periods.
• Data minimisation: send the model only what the task needs. Strip identifiers before processing where you can.
• Records and DPIA: add the AI processing to your records of processing activities, and assess whether a data protection impact assessment is needed.
• AI Act transparency basics: people should know when they are interacting with an AI system unless it is obvious, and AI-generated content may need to be identified in some contexts. High-risk uses such as certain employment or credit decisions carry much heavier obligations.

A cost comparison framework

We deliberately do not publish precise prices here, because GPU rental and API pricing change quickly. Use this framework with current quotes.

Assumptions to make explicit in your own comparison: monthly token volume (input and output separately), peak concurrency, required model quality, and the internal hourly cost of whoever maintains the system.

The break-even logic is simple: estimate your monthly API bill at realistic volume, then compare it with the fixed server cost plus maintenance time. If the API bill is small, self-hosting rarely wins on cost alone, and the decision should rest on data control. If volume is high and steady, dedicated hardware can be much cheaper per request. We explain the broader economics in own EU hardware vs hyperscalers.

What we recommend in 2026

1. Start with the use case and an evaluation set, not with hardware.
2. Prototype on an EU-hosted API to validate value quickly.
3. Design the application against an OpenAI-compatible interface, so switching to a self-hosted vLLM endpoint is a configuration change.
4. Move to self-hosting when data requirements or volume justify it, on EU hardware you control.

Our custom AI development team builds these systems end to end, and our EU hosting runs them on dedicated hardware in Germany with flat pricing. If you are not sure which tier fits, start with consulting.

Want a clear answer for your own situation? Send us your use case and we will return a free project roadmap within 24 hours, including a recommendation on API versus self-hosted.