Upload a document?
Production ingest is not “upload → embed”. Watch parse, clean, metadata, and structure-aware chunking — then go to Step 2 · Ask & test.
Production RAG map?
Follow this while you upload and ask — every live panel maps to a box here.
01
Data Sources
PDF · DB · API · Web
02
Ingestion pipeline
Parse · Clean · Chunk · Metadata · Versioning
03
Embedding + indexing
Vectors + keyword tokens
04
Dual stores
Qdrant + BM25 keyword index
05
User query path
Rewrite → filters → hybrid → rerank → compress → LLM → citations
06
Eval + observability
Grounding · feedback · latency traces
Query path (playground)
Query → rewrite → metadata filter → hybrid (Qdrant + BM25) → rerank → context compression → LLM → citations → answer → eval / observability
Hybrid retrieval: We run vector search and keyword/BM25 search, then fuse ranks (RRF) so meaning and exact terms both win.
Ingestion deep-dive?
Parsing
Bytes → pages, headings, tables, images
- ·OCR (Mistral) for PDFs and images, including scans
- ·Preserve layout signals (markdown headings, page breaks)
- ·Detect tables/images as first-class objects when present
Live: watch the matching card in the pipeline below when you upload.
Cleaning
Remove noise that hurts retrieval
- ·Drop repeated headers/footers and near-duplicate lines
- ·Normalize whitespace without destroying tables
- ·Keep a before/after sample so you can audit cleaning
Live: watch the matching card in the pipeline below when you upload.
Metadata extraction
Labels for filter + permissions later
- ·Doc type, language, version, date, department tags
- ·Enables metadata filtering at query time
- ·Foundation for access-control filters in production
Live: watch the matching card in the pipeline below when you upload.
Structure-aware chunking
Split on structure, not only token count
- ·Prefer heading / page / table boundaries
- ·Overlap only inside long unstructured sections
- ·Each chunk keeps section title + strategy tag
Live: watch the matching card in the pipeline below when you upload.
How RAG works?
1
Ingest with structure
Parse, clean, extract metadata, then structure-aware chunk — not a black-box upload.
2
Index two ways
Store meaning vectors in Qdrant and keep keyword signals for hybrid search.
3
Retrieve like production
Rewrite the question, filter metadata, fuse vector+BM25, rerank, then compress context.
4
Ground and verify
Generate with citations, then test faithfulness with in-doc and out-of-doc probes.
Drop knowledge sources here
PDF & images use Mistral OCR · also MD, DOCX, CSV
Real upload · OCR runs on the server with your Mistral key / free key
Your knowledge base?
Live pipeline?
Upload or select a source to watch the live ingestion pipeline — parse, clean, chunk, embed, and index in real time.