Not long ago, running a capable language model on your own hardware was a research project. Today a single well-specified server can host models that handle most everyday business tasks well. For organisations that hold sensitive data, that changes the question. It is no longer whether to use AI, but where to run it and under whose control.
What private AI means, and when it is worth it
- Client, patient, employee or citizen data cannot leave the organisation, by contract, by regulation or by policy
- Auditors or regulators ask where data is processed and who can see it, and you want a short answer
- Usage-based pricing grows with every user and every document, and you expect broad adoption
- You want to decide when the model changes, rather than discovering a change after answers shift
- The assistant must reach internal systems that should never face the internet
In a private AI system, the model, the documents it reads and every conversation stay inside infrastructure you control. That can be servers in your own building or data centre, or a private cloud environment dedicated to you. Nothing is sent to a public AI service, and nobody outside the organisation can use your data to train their models.
Access follows the roles you already have, and every question and answer can be logged. Private AI is not right for everyone: if your data is not sensitive and volumes are small, a reputable public service with a proper data processing agreement may be simpler. It earns its place when one or more of these is true.
How good are open models?
For answering questions from documents, drafting, summarising, translating and classifying, today's best open-source models are close to the leading public services, and the gap has narrowed with each generation. For most internal assistants, the quality of the documents and the retrieval matters more than the last few points of model capability.
Where a task genuinely needs the strongest available reasoning, a hybrid design helps. Sensitive documents, personal data and internal systems stay inside, while clearly non-sensitive work, such as rewording a public web page, may go to a public model. A routing layer decides where each request goes and logs the decision, so the rule is enforced by the system rather than left to each user.
The model is the easy part. The value, and the risk, sit in what it is allowed to read and who is allowed to ask.
Hardware and sizing in plain terms
The main constraint is memory on the graphics processors (GPUs). A larger model needs more GPU memory to load, and more people asking at once needs more headroom. Techniques such as quantisation, which stores the model in a more compact form, let a given server run larger models with a small loss in quality.
In practice, sizing comes down to three questions. How many people will use it at the same time, not in total? How long are the documents and answers? How fast must a reply appear? A team of a few dozen people asking occasional questions can often be served by a single server with one or two modern GPUs. Hundreds of concurrent users, long documents or several models in parallel call for more.
If buying hardware feels premature, start in a private cloud environment in the region you need, measure real usage, then decide. A well-designed system moves onto your own servers without being rebuilt.
Retrieval with role-based access
Most of the value in a private assistant comes from retrieval: the system finds the relevant passages in your documents and the model answers from them, citing its sources. This is what turns a general model into one that knows your policies, contracts and procedures.
Access control has to apply at retrieval time, not just at login. Each document, or each section, carries the permissions of its source system. When someone asks a question, the search only looks at what that person is allowed to see. If it is designed this way from the start, the assistant cannot leak a document through a summary.
Evaluate before people rely on it
An assistant that is right most of the time and confidently wrong occasionally is a risk, not a tool. Before rollout, build a test set: real questions from the people who will use it, with the correct answers and sources agreed by someone who knows the subject. Run every model and configuration change against that set.
Measure what matters to the business: is the answer correct, is the source right, does it refuse when it should, and does it say when it does not know. Keep the test set alive after launch by adding questions that users flag as wrong. On an enterprise assistant we built for more than 2,000 staff, this routine was what gave leadership the confidence to widen access.
GDPR and the EU AI Act
- Documentation: what the system does, which models it uses, what data it reads and how it was tested
- Transparency: users know they are talking to an AI and can see the sources behind each answer
- Human oversight: a person reviews outputs before they affect a decision about someone
- AI literacy: staff who use or run the system understand its limits, which the Act asks of deployers
Keeping data inside your own infrastructure simplifies GDPR considerably. There are no international transfers to assess, no new processors to add to your records, and a clear log of who saw what. You still need a lawful basis for the processing, a record of it, sensible retention rules for conversation logs, and in many cases a data protection impact assessment.
The EU AI Act takes a risk-based approach. Some practices are prohibited outright. Systems used in sensitive areas listed in the Act, such as recruitment, staff evaluation or access to essential services, are high-risk and carry the heaviest obligations. Systems that interact with people carry transparency duties, so users must know they are dealing with AI. Most internal knowledge assistants sit in the lower categories.
Even then, good practice covers more than the legal minimum, and it makes later audits straightforward.
A first 30, 60 and 90 days
Most of the mistakes we see are avoidable: starting with every document in the organisation, treating permissions as a login problem, skipping evaluation because early demos looked good, buying hardware before measuring real use, and having no owner after launch.
- Days 1 to 30: scope and foundationsChoose one team, one body of documents and one clear job for the assistant. Agree the data rules, roles and success measures. Set up the environment, load the documents with their permissions, and build the first test set with the team.
- Days 31 to 60: pilot with real usersGive the assistant to a small group doing real work. Review answers weekly against the test set, fix retrieval gaps and document quality issues, and adjust the model choice if needed. Write the documentation as you go, not at the end.
- Days 61 to 90: harden and decideAdd monitoring, logging and a process for updating models. Train users on what the assistant is for and where its limits are. Then decide, on evidence, whether to widen access, add a second use case, or move from private cloud to your own servers.
Questions we are often asked
Do we need our own data centre?
No. Many organisations start in a private cloud environment in the region they need and only move onto their own servers once usage is proven. The design is the same in both cases.
Will the model learn from our data and share it?
Not in a private deployment. The model reads your documents at the moment of answering; it is not retrained on them, and nothing leaves your infrastructure.
How often should we update the model?
When a newer model clearly beats the current one on your own test set, not on the calendar. Because you control the system, you decide the timing and can roll back if needed.
Is a private assistant high-risk under the EU AI Act?
Usually not, if it answers questions from internal documents. It can become high-risk if it is used for decisions in areas the Act lists, such as recruitment or evaluating staff, so the use matters more than the technology.