On-premise AI: running a private LLM on your own infrastructure.

Why companies choose on-premise AI

• Regulation or contracts that restrict where data may be processed
• Intellectual property, source code and customer data that must not leave the network
• Air-gapped or restricted environments with no internet access
• Predictable cost at high usage, instead of a bill that grows with every request

Three deployment options

• Public API: the fastest to start, and the one that security teams most often refuse for sensitive data.
• Managed model in your own cloud tenancy: models such as Claude used through your own cloud account, under your own access and data controls. Strong models without operating GPUs yourself.
• Self-hosted open model: an open model running on your own GPU servers. Nothing leaves your network, and you carry the operational work.

What the architecture looks like

• GPU servers, on bare metal or as Kubernetes nodes
• A model serving layer that loads the model and answers requests
• A search or vector index over your documents, for retrieval-augmented generation (RAG)
• A gateway in front, handling authentication, rate limits and logging
• Monitoring for latency, GPU utilisation, quality and errors
• Backups and an update process for models and software

How to size the hardware

The right hardware depends on three things: the size of the model, how many people use it at the same time, and how fast answers need to be. A common mistake is buying for the largest model before knowing whether a smaller one answers your questions well enough.

Start with a pilot on a single GPU server and one real use case. Measure answer quality and response time with real users, then decide what to buy.

RAG or fine-tuning?

For internal knowledge, retrieval-augmented generation is almost always the first step. Your documents are indexed inside your environment, and the assistant looks up the relevant passages when it answers. New documents are available as soon as they are indexed, and access rights can follow the source systems.

Fine-tuning changes the model itself. It is useful for a specific style or task, costs more to maintain, and does not replace retrieval for facts that change.

Security checklist

• The assistant only retrieves what the asking user is allowed to see
• Data is encrypted in transit and at rest
• Every request is logged, with a retention period you have chosen
• The AI environment is isolated from the rest of the network
• In air-gapped environments, models and packages come from an internal mirror