Skip to content

GitLab

Merged
Created by Satyam Raj@satyamrajMaintainer

Harden KB retrieval/ingest and add split-host deploy tooling

Knowledge Base:

  • Replace naive TF sparse vectors with a real BM25 model: IDF-weighted document vectors (K1/B, length-normalized) and presence-indicator query vectors, persisted to data/bm25.json and reloaded by incremental sync. Colliding hashed indices are summed to avoid Qdrant 422s on duplicate indices.
  • Make the full ingest resumable: phase 1 persists the chunk set + BM25 model, phase 2 records an upsert cursor after each batch, so an interrupted multi-hour embed resumes instead of restarting. npm run reindex auto-resumes; --fresh forces a clean rebuild.
  • Retry embeddings with backoff on transient Ollama failures (sleep/timeout/5xx) so a long CPU-bound run self-heals.
  • Embed with nomic task prefixes (search_query/search_document) so query and document vectors share a space; toggle via EMBED_PREFIX_ENABLED.
  • Fix chunker heap-death on minified/single-line files (require a break past the window midpoint) and prepend the SOURCE header to every file chunk, not just the first, so all chunks retain project/file/URL provenance.
  • Cap oversized blob downloads and skip them (generated/minified files).
  • Put /api/stats and /api/recent behind requireAuth; default RERANK_MODEL off (stock Ollama has no /api/rerank).

GitLab MCP:

  • Register kb_* tools only when KB_URL is set, so no dead schema ships when the Knowledge Base is absent.
  • Cap read_file output (12k whole-file, 40k ranged) to protect the context window.
  • Drop the kb_ask reference from user_commits' description.

Deploy:

  • Add deploy/mcp-only/ (compose + env template + runbook) for the MCP-only box.
  • Add ARCHITECTURE-SPLIT / DEPLOYMENT-IDEATION docs and docker-compose.dev.yml for isolated local Qdrant/Ollama.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com