Hundreds of startups have popped up offering “turnkey AI chatbots for your documentation” for $99 to $299 per month.
These SaaS platforms promise simple one-click setups, but they come with serious drawbacks for solo developers:
- Expensive Per-Query Markups: They charge 10x markups on raw OpenAI or Anthropic API token costs.
- Complete Vendor Lock-In: Your chunking configurations, vector embeddings, and retrieval telemetry are trapped in a proprietary black box.
- Rigid Customization: If you want to customize search re-ranking, filter by customer subscription tier, or integrate custom markdown syntax, you hit a hard wall.
Building your own production-grade RAG (Retrieval-Augmented Generation) pipeline for documentation is surprisingly straightforward and costs less than $10 a month to run.
Here is the clean, open architecture I use to build self-hosted RAG knowledge bases without third-party vendor lock-in.
Core Architecture: The 3 Components
A clean, maintainable RAG system requires only three foundational tools:
- Embedding Model: OpenAI
text-embedding-3-small(costs $0.02 per 1M tokens—essentially free). - Vector Database: PostgreSQL with the
pgvectorextension (available on Supabase, Neon, or standard RDS). - Generative LLM: Claude 3.5 Haiku or GPT-4o-mini for rapid, accurate question answering.
By storing vectors directly inside your existing PostgreSQL database, you eliminate the need for specialized vector-only databases (like Pinecone or Qdrant), keeping your infrastructure unified in one place.
Step 1: Document Chunking Strategy
The secret to accurate RAG is not the LLM; it is chunking quality.
Do not split documentation by arbitrary character counts (e.g., splitting every 500 characters). Arbitrary splits sever code snippets in half and orphan explanatory paragraphs from their relevant headings.
Instead, chunk by Markdown Section:
- Parse documentation files by H2 (
##) and H3 (###) headers. - Prepend the page title and breadcrumb hierarchy to the top of every chunk:
Document: Billing & Invoicing > Updating Credit Cards Section: How to handle expired payment methods Content: [Section text and code snippet]
Prepending contextual metadata ensures that the embedding vector captures the overarching topic even if the isolated paragraph contains brief text.
Step 2: PostgreSQL Schema with pgvector
Enable the vector extension and create a simple documentation embeddings table:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE doc_chunks (
id SERIAL PRIMARY KEY,
doc_path TEXT NOT NULL,
section_title TEXT NOT NULL,
content TEXT NOT NULL,
metadata JSONB DEFAULT '{}'::jsonb,
embedding vector(1536)
);
-- Create an HNSW index for sub-10ms vector similarity searches
CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);Step 3: Similarity Search and Context Retrieval
When a user asks a question, generate an embedding for their query using text-embedding-3-small, then query PostgreSQL using the cosine distance operator (<=>):
SELECT content, section_title, 1 - (embedding <=> $1::vector) AS similarity
FROM doc_chunks
WHERE 1 - (embedding <=> $1::vector) > 0.75
ORDER BY embedding <=> $1::vector
LIMIT 4;In less than 15 milliseconds, PostgreSQL returns the 4 most semantically relevant documentation snippets.
Step 4: Grounded Prompt Assembly
Feed the retrieved snippets into your generative model with a strict grounding constraint:
You are the StartupTrio documentation assistant.
Answer the user's question using ONLY the provided documentation excerpts below.
If the answer cannot be found in the documentation, reply:
"I cannot find this in our documentation. Please contact support."
<documentation_context>
[Retrieved Chunks 1 to 4]
</documentation_context>
User Question: [User Query]By explicitly restricting the model to the provided context, you eliminate hallucinations and provide instant, accurate technical answers to your users at a cost of roughly $0.001 per query.
Related Operational Guides
For more engineering and AI architecture playbooks, see: