Building a RAG System — Make Your Own Documents Answer Questions

A client has a 400-page procedures manual and an employee asks 'how do I submit a travel expense report?'. A human finds the answer in five minutes. A RAG system finds it in two seconds. This post is the playbook I extracted after shipping four RAG systems in the last six months — what worked, what didn't.

RAG in one paragraph

Retrieval Augmented Generation. The idea: every time you query an LLM, you also pull the most relevant chunks from your own knowledge base and pass them along. The LLM reads from those chunks to answer. The model becomes aware of your specific data without fine-tuning, hallucinations drop, updates are trivial.

Architecture step by step

• Ingest documents — PDFs, DOCX, web pages, Confluence, whatever
• Chunk into pieces — 400-800 token chunks with overlap
• Embed each chunk — turn each into a vector (OpenAI text-embedding-3-small works fine)
• Store in a vector DB — pgvector, Qdrant, or Pinecone
• On query — embed the question, pull the 5-8 nearest chunks
• Send to LLM — context + question, read the answer

Chunking — where most people get it wrong

Slicing at every 500 characters is a disaster — sentences cut in half, context gone. I always do semantic chunking: split by paragraphs first, then by sentences if paragraphs are too long, keep 50-100 tokens of overlap between adjacent chunks. Crucially, prepend the markdown heading path so the model knows the chunk belongs under 'Salary calculation' or wherever.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=80,
    separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
    length_function=lambda x: len(tokenizer.encode(x)),
)

# Markdown başlıqlarını chunk-a daxil et
def chunk_with_headings(doc):
    sections = split_by_heading(doc)
    chunks = []
    for heading, content in sections:
        for chunk in splitter.split_text(content):
            chunks.append(f"# {heading}\n\n{chunk}")
    return chunks

Vector DB — I love pgvector

Pinecone, Weaviate, Qdrant — they all work. My default is pgvector: it's a PostgreSQL extension that runs on your existing Postgres, no separate service to deploy. Performance is excellent up to a million vectors. Past 10M I'd move to Qdrant.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id BIGSERIAL PRIMARY KEY,
  document_id TEXT NOT NULL,
  content TEXT NOT NULL,
  embedding vector(1536),
  metadata JSONB,
  created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON chunks USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

-- Ən yaxın 8 chunk
SELECT content, metadata
FROM chunks
ORDER BY embedding <=> $1
LIMIT 8;

Embedding model — which one

OpenAI text-embedding-3-small (1536 dim) is the right default: $0.02 per 1M tokens, fast, multilingual (it handles Azerbaijani fine). The large variant (3072 dim) is slightly more accurate but 6× the price. Specialised domains like medical might benefit from a fine-tuned model, but 90% of projects don't need it.

Hybrid search — vector + keyword

Pure semantic search can miss exact-match queries. If someone searches 'Q4 2025 budget', semantic search might return 'fiscal planning' but skip the exact phrase 'Q4 2025'. Fix: hybrid search. I add a tsvector column in Postgres, run both vector and full-text in parallel, and merge results with Reciprocal Rank Fusion.

Re-ranking — the final polish

Initial retrieval returns 20 chunks. Sending all 20 to the LLM is expensive and triggers 'lost in the middle' (models forget what's in the middle of a long context). I use Cohere's rerank-multilingual-v3.0 to re-score the 20, then take the top 5. Answer quality improves visibly with this one step.

Prompt — where RAG lives or dies

Sən {şirkət adı} köməkçisisən. Yalnız aşağıdakı kontekst əsasında cavab ver.

KONTEKST:
{retrieved_chunks}

QAYDALAR:
1. Cavab kontekstdə yoxdursa, "Bu sualın cavabı bilik bazasında tapılmadı" yaz.
2. Heç vaxt kontekstdən kənar məlumat uydurma.
3. Cavabın hansı sənəddən gəldiyini sonda göstər: [Mənbə: filename.pdf]
4. Cavabı qısa və konkret saxla (maks 3 abzas).

SUAL: {question}

What I actually use

• Embedding: OpenAI text-embedding-3-small
• Vector DB: Postgres with pgvector
• Reranker: Cohere rerank-multilingual-v3.0
• LLM: GPT-4o-mini for 90% of queries, 4o only for complex synthesis
• Framework: No LangChain — plain Python is easier to debug
• UI: Next.js with streaming responses

If you need a RAG system at your company — internal knowledge base, customer support, document Q&A — get in touch and we'll do a 30-minute discovery call on your specific use case.