SkillVaultskills Browse all 500 skills

Search · Version 1.5.0 · Reviewed 2026-08-02

Search Relevance Tuner

Make a defensible decision about ranking experiment design and analyzer tuning with evidence, explicit trade-offs, and a verification plan.

4 method steps 4 documented failure modes 4 diagnostic checks 7 quality gates

Improves lexical and semantic search using judged queries, analyzers, field boosts, synonyms, freshness, popularity, and explainable ranking experiments.

₹99 one-time

Get this skill archive

What this skill helps you do

  • Ranking experiment design
  • Analyzer tuning
  • Zero-result reduction

How Search Relevance Tuner works

You provide

Index config, analyzers, and queries that return the wrong thing

It inspects

Recall before ranking for ranking experiment design

It decides

A analyzer tuning change scoped to the failing query class

You verify

Recall@k and NDCG re-measured on a judged query set

What it checks first

Search Relevance Tuner improves lexical and semantic search using judged queries, analyzers, field boosts, synonyms, freshness, popularity, and explainable ranking experiments. Use it when the work involves Ranking experiment design, Analyzer tuning, Zero-result reduction.

  1. Whether relevance failures are recall problems or ranking problems, since the fixes are unrelated.
  2. Query analysis and index analysis symmetry, because a mismatch silently breaks exact matching.
  3. Whether rare tokens such as identifiers and codes are preserved rather than stemmed away.
  4. Click and abandonment data, which reveals relevance failures no offline metric captures.

Failure modes it recognizes

  • Aggressive stemming or lowercasing destroying exact matches on part numbers and error codes.
  • A relevance tweak that improves head queries while breaking the long tail nobody measures.
  • Synonym expansion introducing false positives that outrank the exact match.
  • Analyzer changes applied at query time but not reindexed, so the two sides disagree.

Answers it will reject

  • Tuning ranking weights before confirming the correct document is retrieved at all.
  • Judging relevance from a handful of favorite queries rather than a sampled distribution.
  • Boosting recency or popularity globally, which buries precise matches for specific queries.

Decision rules it applies

  • Fix recall before ranking; a document that is not retrieved cannot be ranked.
  • Keep query-time and index-time analysis identical, and reindex when either changes.
  • Evaluate on a judged set that includes head, torso, and tail queries.

Evidence it asks for

  • Measure recall@k and NDCG on a labeled query set before and after each change.
  • Track zero-result and reformulation rates as leading indicators of relevance failure.
  • Inspect the analyzed form of both the query and the document when a match fails.

The method inside

  1. Establish what is actually true about ranking experiment design from the supplied evidence, and mark what is missing.
  2. Identify the mechanism behind analyzer tuning rather than restating the symptom.
  3. Choose the smallest defensible change for zero-result reduction, weighing impact, confidence, effort, and reversibility.
  4. Record consequences, rollback, and open questions

Deliverables

  • Ranking experiment design assessment
  • Analyzer tuning decision and action plan
  • Zero-result reduction verification checklist

Evidence requirements

  • Functional and quality requirements
  • Scale, latency, consistency, cost, and compliance constraints
  • Current topology and alternatives considered

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

Our product search returns technically matching but commercially irrelevant results. Create a relevance-tuning plan.

Expected output

Build a judged query set from real sessions, separate retrieval recall from ranking, inspect title/category field contributions, and test business signals as controlled features rather than hardcoded boosts...

Boundaries and compatibility

Ideal for

  • Ranking experiment design: produce a decision or artifact grounded in supplied evidence.
  • Analyzer tuning: produce a decision or artifact grounded in supplied evidence.
  • Zero-result reduction: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Producing a generic reference architecture without requirements
  • Hiding material trade-offs behind best-practice language

Agent compatibility

  • GitHub Copilot custom agents
  • Claude Agent Skills / SKILL.md
  • Any instruction-following chat model

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.