How to Set Up a Local RAG — Chat With Your Own Documents (2026)

RAG (retrieval-augmented generation) is how you get a local model to answer questions about your own PDFs, notes, and docs — without fine-tuning and without sending anything to the cloud. It works by converting your documents into vectors with an embedding model, finding the passages most relevant to your question, and feeding them to the chat model as context. This guide sets it up locally, privately, the straightforward way.

If you have not run a local model yet, start with how to install Ollama and best local LLM models 2026.

What You Need

• Ollama with a chat model pulled (e.g. ollama run llama3.1:8b once).
• An interface that does RAG: Open WebUI (built-in document chat) or AnythingLLM (a dedicated RAG app). Both keep everything local.
• An embedding model — this is the piece beginners miss. RAG needs a model that turns text into vectors:
  • nomic-embed-text — the popular default, fast and solid.
  • bge-m3 — better multilingual support and larger context.
• Your documents: PDF, DOCX, TXT, MD, CSV — whatever you want to query.

Step-By-Step Setup

1. Pull a chat model and an embedding model

ollama pull llama3.1:8b
ollama pull nomic-embed-text

The embedding model is separate from the chat model and is what makes retrieval work.

2A. Option A — Open WebUI (simplest)

In Open WebUI, open Settings → Documents:

• Set the embedding engine to Ollama and the embedding model to nomic-embed-text.
• Set a sensible chunk size and overlap (e.g. ~1000–3000 / ~100) so documents are split into retrievable pieces.

Then click the paperclip in a chat, upload a PDF or text file, and ask about it. Open WebUI chunks, indexes, and retrieves automatically.

2B. Option B — AnythingLLM (dedicated RAG)

Open AnythingLLM (default http://localhost:3001), create a local admin account, then:

• Settings → LLM Provider → Ollama, pick your chat model.
• Settings → Embedding Provider → Ollama, pick nomic-embed-text.
• Create a workspace, upload your files (PDF, DOCX, TXT, MD, CSV, JSON, code), and click Save and Embed to index them.

Ask a question and AnythingLLM retrieves the relevant passages and feeds them to the model.

3. Fix the context-length gotcha

This is the single most common reason local RAG gives weak answers: Ollama defaults to a 2048-token context window, which is too small to hold retrieved passages plus your question. Raise the model’s context length (the num_ctx setting in your UI, or a custom model with a larger context) so the retrieved chunks actually fit. Match it to your hardware — longer context uses more VRAM (see the VRAM requirements guide).

4. Ask and verify

Ask a question whose answer is in your documents, and check that the response cites or reflects the source text. If answers ignore the documents, the embedding model or context length is usually the culprit (see Common Mistakes).

Common Mistakes

• No embedding model. RAG cannot work without one — pull nomic-embed-text (or bge-m3) and select it in the tool’s embedding settings.
• Leaving Ollama at the default 2048-token context. Retrieved passages get truncated and the model “forgets” them. Raise the context length.
• Chunks too big or too small. Huge chunks dilute retrieval; tiny chunks lose meaning. Start around 1000–3000 characters with light overlap and tune.
• Expecting RAG to replace a bigger model. RAG adds your knowledge; it does not make a small model smarter at reasoning. Pair it with a capable chat model that fits your VRAM.

What To Do Next

• Pick a strong chat model for the generation step: best local LLM models 2026.
• If you have not set up the interface yet: how to set up Open WebUI.
• Long documents need context headroom (and VRAM) — confirm with the VRAM calculator; a 24GB card helps for big knowledge bases.
• Care about keeping data private? That is the whole point — see local AI vs ChatGPT cost for the broader case for running locally.

Frequently Asked Questions

What is a local RAG?

RAG (retrieval-augmented generation) lets a local model answer questions about your own documents. It embeds your files into vectors, retrieves the passages most relevant to your question, and feeds them to the chat model — all on your hardware, with nothing sent to the cloud.

Do I need a separate embedding model for RAG?

Yes. RAG needs an embedding model (separate from the chat model) to turn text into vectors for retrieval. Pull nomic-embed-text with Ollama, or bge-m3 for better multilingual support, and select it in your tool’s embedding settings.

Why does my local RAG give poor answers?

The two usual causes: no embedding model selected, or Ollama’s default 2048-token context being too small to hold retrieved passages. Add an embedding model and raise the context length. Poor chunking (too large or too small) is the next thing to tune.

Open WebUI or AnythingLLM for RAG?

Open WebUI has built-in document chat and is the simplest if you already use it for chat. AnythingLLM is a dedicated RAG app with workspaces and broad file support. Both run locally on top of Ollama — pick whichever fits your workflow.

Is a local RAG private?

Yes. With Ollama plus Open WebUI or AnythingLLM, your documents, embeddings, and chats stay on your machine — nothing is uploaded. That privacy is the main reason to run RAG locally instead of using a cloud service.

---

Last updated: June 2026. Setup steps reference current Open WebUI and AnythingLLM releases and Ollama embedding models. We do not publish invented benchmarks.