Creating an In-House LLM: Company Knowledge Instead of Hallucinations

The decision is made: the company is getting its own in-house LLM, essentially a private version of ChatGPT that every employee can rely on without worrying about invented answers. On paper, that sounds like the cleanest solution there is.

In practice, it's a full infrastructure project, with everything that comes with it: a base model, your own server operations, and technology that makes sure the system really only answers from company knowledge. This article looks at what's actually behind it, where the limits are, and how companies approach the topic in practice.

What an Internal LLM Actually Is

An internal LLM is a language model that a company hosts and operates itself, on its own servers or in a dedicated, secured private cloud. By connecting it to internal sources, such as manuals, SOPs, internal wikis, training materials, or product documentation, its answers can be deliberately grounded in that content. Certain areas can also be deliberately excluded. The data and processing run entirely within the company's own infrastructure, not through the servers of a third party provider such as OpenAI, Anthropic, or Google. "AI assistant" is the umbrella term here: it covers both self hosted internal LLMs and managed service solutions like the Elephant AI assistant.

How an Internal LLM Is Technically Built

It typically takes three building blocks:

1. A base model: Usually open source or source available, such as Llama, Mistral, or Qwen. Most variants can be used by companies free of charge, but depending on the model and company size, usage limits or commercial license terms may apply and should be checked in advance. Llama, for example, permits commercial use free of charge, unless the company itself, including affiliated companies, has more than 700 million monthly active users across its own products, in which case a separate license from Meta is required. The license also excludes certain use cases and, starting with Llama 3, requires that any fine tunes carry "Llama" in their name. For Mistral, most of the open models are freely usable under Apache 2.0; individual models such as Devstral 2 123B require a paid license once the using company's worldwide monthly group revenue exceeds 20 million US dollars. For Qwen, the smaller variants are mostly Apache 2.0; older large models like Qwen 72B had their own license with a cap of 100 million monthly active users. The current Qwen3 generation is consistently under Apache 2.0, including the large variants.

2. Owned or rented server infrastructure: With enough compute power to run the model.

3. A retrieval architecture (RAG): It first matches questions against the relevant internal content, the so called company knowledge, before the model formulates an answer from it. Instead of generating an answer directly from the model's trained memory, the system first searches for the passage that fits the question in meaning, not just in matching words, and only then passes that retrieved passage along with the question to the model.

On top of that comes ongoing operations: model updates, security patches, access rights, scaling as usage grows. Technically feasible, but an ongoing IT effort that many companies don't want to, or can't, take on.

ChatGPT or Claude as an Alternative to an Internal LLM: Possible, But Effortful

ChatGPT and other options like Claude or Gemini work very well for simple queries and low complexity topics. Technically, more can also be connected than you might initially think: Google Drive or Confluence can also be linked up in ChatGPT, and with a Business or Enterprise license, inputs aren't used to train the model. So the difference from a dedicated AI assistant isn't that ChatGPT fundamentally can't do this.

The real difference lies in the effort involved and in consistency:

Individual configuration instead of central configuration. With ChatGPT or Claude, each person has to decide for themselves which knowledge to answer from, and use their own prompts or custom instructions to make sure the model cites only from that knowledge. That works, but it depends on every individual's diligence, and it's hard to keep consistent across a larger team. A dedicated AI assistant, by contrast, is set up centrally once and automatically applies equally to all employees, without anyone having to configure or prompt anything themselves.

So switching to a dedicated solution doesn't necessarily happen because ChatGPT is technically limited, but because consistency, governance, and analytics can be mapped centrally, without any configuration effort for individual employees.

The Elephant AI Assistant as an Alternative to an Internal LLM

The Elephant AI Assistant pursues the same goal as an internal LLM, source based answers drawn from company knowledge, but achieves it without its own infrastructure. Instead of hosting a model itself, the retrieval logic runs as a managed service: upload documents or connect sources like SharePoint, Confluence, or Google Drive, and the rest runs automatically. Connected sources sync automatically every 24 hours. If an SOP changes on SharePoint, the updated version is available in the assistant with no manual effort. The key differences from a classic internal LLM are simplicity instead of infrastructure overhead, plus two safety features: answers come exclusively from approved company knowledge, and missing sources are openly flagged rather than the system inventing an answer. Every answer also links to the original passage in the source document, so employees can verify for themselves where the information came from. Technically, this is also possible with a self hosted internal LLM, but there it first has to be developed and maintained in house rather than being built in from the start. That doesn't make the assistant an internal LLM in the classic sense, but rather a simpler alternative for companies that want the same benefits without hosting and operating models themselves.

Relieving Team Leads, Especially in the First Few Months

The biggest impact of an AI assistant like Elephant's shows up not with experienced employees, but with new ones. Anyone who has only been on the job for a few weeks or months naturally has the most follow up questions, and it's exactly these questions that usually end up with the team lead, because they're the fastest person to ask.

Masijar Khedmatgozar, Senior Team Lead at Enpal, describes the effect this way: "The AI assistant optimizes our work processes by answering routine questions quickly and precisely. This gives us as managers more time for our core tasks, without being interrupted by frequent queries."

In the case of Enpal Montage GmbH, a multilingual field service team, Elephant documents concrete effects for onboarding: 30 percent faster ramp up and 40 percent less in person training, with over 9,000 questions per month that the AI assistant answers directly, without a team lead having to get involved.

The Elephant AI Assistant in Everyday Use

Concretely, the simpler alternative looks like this:

• Questions can be asked by text, voice message, or photo, for example a photo of an error code on a machine. Answers come back in over 95 languages, regardless of which language the source document is in, which is especially relevant for teams with many nationalities. Common formats such as PDF, DOCX, PPTX, XLSX, CSV, and video are supported.
• Every question asked is also evaluated as a data point. The built in question analysis shows which topics come up frequently and where the assistant couldn't deliver a satisfactory answer, filterable by team, time period, and feedback type. That makes knowledge gaps visible before they turn into real problems in the field, and it's considerably more precise than a retrospective employee survey.
• The platform is hosted in Germany, secured through a data processing agreement under Art. 28 GDPR.

What to Look for When Choosing an AI Assistant

Not every tool that calls itself an AI assistant actually delivers the source fidelity of a well implemented internal LLM. A few questions worth asking before deciding:

• Does the system answer exclusively from your company knowledge, or does it unnoticeably blend in free knowledge from the internet? Only the first approach rules out hallucination from external, unverified sources.
• What happens when no matching source is found? Does the system say so openly, or does it still produce an answer, just a guessed one?
• Can the tool connect to your existing sources, such as SharePoint, Confluence, or Google Drive, and how often does it sync? Without automatic syncing, someone ends up manually maintaining two systems in parallel after all.
• Where is the data processed, and is there a GDPR compliant data processing agreement under Art. 28? That's one of the decisive questions on data sovereignty, alongside the server location itself.
• Can access rights be restricted per team, so each team only sees the knowledge relevant to its own work?
• Does the tool show where knowledge is missing? A question analysis reveals which topics come up frequently and where no good answer exists, which is the real lever for better documentation.
• Does the system support the languages and file formats your team actually needs, and is there a feedback loop that allows answers to be continuously improved?

But the most important point is this: ultimately, answer quality depends directly on the quality and currency of the connected documents anyway. A system built on outdated SOPs delivers outdated answers, no matter how good the underlying technology is.

Conclusion

An internal LLM in the true sense, self hosted, run on the company's own servers, solves a very concrete problem: employees should be able to get to company knowledge quickly and reliably, without sensitive data leaving the company and without a language model inventing answers out of embarrassment. The price for that is an ongoing infrastructure and IT project. Anyone who wants to achieve this goal without hosting and operating models themselves will find a simpler, more cost effective alternative in solutions like the Elephant AI assistant: source based answers from approved company knowledge, hosted GDPR compliantly in Germany, without a major in house IT project. This isn't an internal LLM in the classic sense, but a pragmatic path to the same benefit, right where the effort in daily work is greatest today: the same recurring questions that would otherwise land on team leads.