SkillVaultskills Browse all 500 skills

AI Engineering · Version 1.2.0 · Reviewed 2026-08-02

RAG Architecture Assistant

Make AI behavior measurable and safer for RAG design and evaluation planning with evidence, explicit trade-offs, and a verification plan.

4 method steps 6 documented failure modes 5 diagnostic checks 7 quality gates

Reviews retrieval, chunking, evaluation, grounding, and production RAG architecture.

₹99 one-time

Get this skill archive

What this skill helps you do

  • RAG design
  • Evaluation planning
  • Retrieval debugging

How RAG Architecture Assistant works

You provide

Chunking, retrieval config, prompts, and failing queries

It inspects

Recall@k before generation, chunk boundaries, vocabulary match

It decides

Whether the fault is retrieval, ranking, or prompt permission

You verify

Labeled recall and claim-level faithfulness scores

What it checks first

RAG Architecture Assistant reviews retrieval, chunking, evaluation, grounding, and production RAG architecture. Use it when the work involves RAG design, Evaluation planning, Retrieval debugging.

  1. Retrieval relevance and answer faithfulness measured separately — conflating them makes the failure undiagnosable.
  2. Whether the correct chunk is present in the retrieved set at all (recall@k), before tuning the generation prompt.
  3. Chunk boundaries relative to semantic units: a fact split across a boundary cannot be retrieved intact by either half.
  4. Whether the query and the document use the same vocabulary, since pure dense retrieval fails on rare exact tokens like error codes and IDs.
  5. Context ordering, because relevant content placed in the middle of a long context is measurably deprioritized.

Failure modes it recognizes

  • High retrieval relevance with hallucinated answers, because the prompt permits synthesis beyond the provided context.
  • Embedding drift after re-embedding with a different model version, silently degrading a previously good index.
  • Chunks lacking document context (title, section, date), so a retrieved fragment is unattributable and ambiguous.
  • Dense-only retrieval missing exact identifiers, product codes, and negations that a lexical index would match.
  • Stale index serving confidently wrong answers with no freshness signal in the response.
  • Retrieved duplicates consuming the context budget and crowding out the one novel chunk that mattered.

Answers it will reject

  • Increasing `k` to fix quality — this dilutes the context and usually reduces faithfulness while raising cost.
  • Evaluating with a handful of hand-picked queries rather than a labeled set containing known-hard and no-answer cases.
  • Treating cosine similarity as relevance without a threshold calibrated on real negatives.
  • Adding a reranker before establishing that the correct document is in the candidate set at all.

Decision rules it applies

  • Fix recall before precision, and precision before prompt wording — the order is not interchangeable.
  • Use hybrid retrieval (lexical plus dense) whenever the corpus contains identifiers, codes, or domain jargon.
  • Require citations at the claim level; if a claim cannot be attributed to a retrieved span, the system must decline.
  • A "no answer in context" response is a correct answer and must be represented in the evaluation set.

Evidence it asks for

  • Measure recall@k and MRR against a labeled query-to-document set before touching generation.
  • Score faithfulness by checking each generated claim against retrieved spans, not against ground truth alone.
  • Track index freshness and embedding model version as first-class metadata on every response.

The method inside

  1. Turn RAG design into explicit functional requirements and quality-attribute constraints.
  2. Model the critical path, state, trust, and failure boundaries that govern evaluation planning.
  3. Compare viable designs for retrieval debugging against weighted constraints and operational ownership.
  4. Select a design with consequences, rollout stages, observability, and a reversible adoption path.

Deliverables

  • RAG design assessment
  • Evaluation planning decision and action plan
  • Retrieval debugging verification checklist

Evidence requirements

  • Prompts, model/version, tools, retrieval path, and examples
  • Evaluation dataset and failure cases
  • Latency, cost, privacy, and policy constraints

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

Why does my support bot retrieve relevant documents but still hallucinate?

Expected output

Separate retrieval relevance from answer faithfulness before changing anything, because they fail independently. Your prompt currently permits unsupported synthesis, so the model fills gaps confidently even when the retrieved passage is correct. Require claim-level citation and an explicit refusal path...

Boundaries and compatibility

Ideal for

  • RAG design: produce a decision or artifact grounded in supplied evidence.
  • Evaluation planning: produce a decision or artifact grounded in supplied evidence.
  • Retrieval debugging: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Treating prompt text as a security boundary
  • Claiming model quality from a handful of demos

Agent compatibility

  • GitHub Copilot custom agents
  • Claude Agent Skills / SKILL.md
  • Any instruction-following chat model

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.