SkillVaultskills Browse all 500 skills

Product Management · Version 1.0.0 · Reviewed 2026-08-02

Experiment Design Reviewer

Make a product decision about power and duration analysis and randomization unit review with evidence, explicit trade-offs, and a verification plan.

4 method steps 6 documented failure modes 5 diagnostic checks 7 quality gates

Reviews experiment design for power, unit of randomization, guardrails, contamination, and the decision rule set before launch.

₹149 one-time

Get this skill archive

What this skill helps you do

  • Power and duration analysis
  • Randomization unit review
  • Guardrail metric design

How Experiment Design Reviewer works

You provide

Hypothesis, baseline rates, traffic, and metrics

It inspects

Required sample, randomization unit, and contamination

It decides

Go or no-go with the decision rule fixed in advance

You verify

Sample ratio checked before any result is read

What it checks first

Experiment Design Reviewer reviews experiment design for power, unit of randomization, guardrails, contamination, and the decision rule set before launch. Use it when the work involves Power and duration analysis, Randomization unit review, Guardrail metric design.

  1. Required sample size for the minimum effect worth acting on, computed before launch rather than after.
  2. The randomization unit and whether it matches the analysis unit, since a mismatch invalidates significance.
  3. Whether the metric is sensitive enough to move within the planned duration at realistic traffic.
  4. Guardrail metrics that would reveal harm the primary metric cannot see.
  5. Whether the decision rule is stated in advance, including what happens on an inconclusive result.

Failure modes it recognizes

  • Peeking at results and stopping when significance appears, which inflates false positives substantially.
  • Randomizing by session while analyzing by user, so returning users appear in both arms.
  • Contamination through shared state, referrals, or a shared account between arms.
  • A novelty effect producing a strong early result that decays within weeks.
  • Sample ratio mismatch indicating a broken assignment mechanism, invalidating the entire result.
  • Running many variants without correction, guaranteeing a spurious winner.

Answers it will reject

  • Choosing duration by calendar convenience rather than by required sample size.
  • Declaring a flat result a success because the new version "is not worse".
  • Segmenting after the fact until a significant subgroup appears.
  • Treating a statistically significant but practically negligible effect as a reason to ship complexity.

Decision rules it applies

  • If the required sample is unreachable in a reasonable window, do not run the test — decide another way and say so.
  • Fix the decision rule, duration, and primary metric before launch and record them.
  • Randomize at the level at which the treatment is experienced, usually the user or account.
  • Check sample ratio mismatch before interpreting any result; a broken split makes the rest meaningless.

Evidence it asks for

  • Compute minimum detectable effect from baseline rate, traffic, and planned duration before launch.
  • Verify assignment balance and guardrail metrics daily without evaluating the primary metric.
  • Report effect size with a confidence interval rather than a binary significance verdict.

The method inside

  1. Separate the customer problem from requested solutions
  2. Inventory assumptions and strength of evidence
  3. Compare options using impact, confidence, risk, effort, and reversibility
  4. Define success, guardrails, and the decision after new evidence

Deliverables

  • Power and duration analysis evidence map
  • Randomization unit review option and risk analysis
  • Guardrail metric design decision memo

Evidence requirements

  • Customer research, usage, support, and commercial evidence
  • Strategy, constraints, dependencies, and opportunity cost
  • Experiment design, roadmap options, or requirements artifact

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

We want to test a new onboarding flow. We will run it for a week and see which converts better.

Expected output

A week is a duration, not a design. Compute the sample needed for the effect size worth acting on: if you cannot reach it, the test will produce a result you should not act on either way. Also confirm randomization is by user rather than session, or returning users will land in both arms...

Boundaries and compatibility

Ideal for

  • Power and duration analysis: produce a decision or artifact grounded in supplied evidence.
  • Randomization unit review: produce a decision or artifact grounded in supplied evidence.
  • Guardrail metric design: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Using request volume as a substitute for impact
  • Presenting a prioritization score as objective truth

Agent compatibility

  • GitHub Copilot custom agents
  • Claude Agent Skills / SKILL.md
  • Any instruction-following chat model

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.