SkillVaultskills Browse all 500 skills

Research · Version 1.3.0 · Reviewed 2026-08-02

Survey Design Reviewer

Produce defensible evidence for question neutrality and sampling validity with evidence, explicit trade-offs, and a verification plan.

4 method steps 4 documented failure modes 4 diagnostic checks 7 quality gates

Reviews surveys for leading questions, response bias, and sampling that invalidates the result.

₹149 one-time

Get this skill archive

What this skill helps you do

  • Question neutrality
  • Sampling validity
  • Scale design

How Survey Design Reviewer works

You provide

Sources, method notes, and the question being answered

It inspects

Evidence strength and negative cases for question neutrality

It decides

A sampling validity synthesis with limitations named

You verify

Findings reproducible from the cited evidence alone

What it checks first

Survey Design Reviewer reviews surveys for leading questions, response bias, and sampling that invalidates the result. Use it when the work involves Question neutrality, Sampling validity, Scale design.

  1. Sample size, selection method, and who was excluded, since these bound every conclusion.
  2. Whether a finding is supported by observed behavior or by what participants said they would do.
  3. Disconfirming cases, which are the fastest test of whether a theme is real.
  4. Whether the question the research answers is the question the decision actually needs.

Failure modes it recognizes

  • Themes derived from the loudest participants rather than from the sampled distribution.
  • Leading questions producing agreement that disappears under neutral phrasing.
  • Survivorship bias from studying only current customers or successful accounts.
  • Coding drift where the same excerpt would be categorized differently later in the process.

Answers it will reject

  • Reporting percentages from a small qualitative sample, which implies precision that does not exist.
  • Treating a synthesized theme as a finding without the excerpts that support it.
  • Stopping at the first pattern that confirms the pre-existing hypothesis.

Decision rules it applies

  • State the sample, the method, and the limitation alongside every finding.
  • Actively seek disconfirming evidence before accepting a theme.
  • Separate what was observed, what was reported, and what was inferred.

Evidence it asks for

  • Attach verbatim excerpts to each theme so a reader can audit the interpretation.
  • Check coding consistency by re-coding a subset and comparing.
  • Report counts with denominators rather than percentages alone.

The method inside

  1. Define the research question and unit of analysis
  2. Create a transparent coding or extraction framework
  3. Preserve source traceability and negative evidence
  4. Separate findings, interpretation, limitations, and applicability

Deliverables

  • Question neutrality evidence table
  • Sampling validity findings with negative cases
  • Scale design limitations and next-research plan

Evidence requirements

  • Source documents, transcripts, data, and research question
  • Sampling method, population, and collection context
  • Known limitations, contradictory cases, and analysis criteria

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

Review this customer survey before we send it. We want to confirm demand for our new feature.

Expected output

Wanting to confirm demand is the risk, and the questions currently do it: each describes the benefit before asking interest. Ask about past behavior rather than future intent, and remove the framing that tells respondents which answer pleases you...

Boundaries and compatibility

Ideal for

  • Question neutrality: produce a decision or artifact grounded in supplied evidence.
  • Sampling validity: produce a decision or artifact grounded in supplied evidence.
  • Scale design: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Fabricating sources, participants, or findings
  • Claiming representativeness without a sampling basis

Agent compatibility

  • GitHub Copilot custom agents
  • Claude Agent Skills / SKILL.md
  • Any instruction-following chat model

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.