SkillVaultskills Browse all 500 skills

Data · Version 1.1.0 · Reviewed 2026-08-02

Late-Arriving Data Advisor

Make data systems more correct and operable for watermark design and restatement policy with evidence, explicit trade-offs, and a verification plan.

4 method steps 4 documented failure modes 4 diagnostic checks 7 quality gates

Designs watermarks, restatement windows, and correction policy for events that arrive after their period closed.

₹99 one-time

Get this skill archive

What this skill helps you do

  • Watermark design
  • Restatement policy
  • Downstream correction

How Late-Arriving Data Advisor works

You provide

Schema, volumes, and the queries that actually run

It inspects

Access patterns and skew affecting watermark design

It decides

A restatement policy design with migration ordering

You verify

Row counts and checksums compared before cutover

What it checks first

Late-Arriving Data Advisor designs watermarks, restatement windows, and correction policy for events that arrive after their period closed. Use it when the work involves Watermark design, Restatement policy, Downstream correction.

  1. The real access patterns, since schema quality is meaningless without the queries it serves.
  2. Cardinality, skew, and growth rate of the key columns.
  3. Consistency and freshness requirements per consumer.
  4. Whether the model preserves history or overwrites it, which determines what questions remain answerable.

Failure modes it recognizes

  • A schema optimized for writes that makes the dominant read pattern a full scan.
  • Skewed partitioning concentrating load on one node.
  • Nullable columns encoding several meanings, making every downstream query ambiguous.
  • A pipeline silently dropping late-arriving records and under-reporting totals.

Answers it will reject

  • Normalizing or denormalizing by preference rather than by measured access pattern.
  • Adding a column for each new requirement until the table encodes multiple entities.
  • Backfilling without a reconciliation check that proves completeness.

Decision rules it applies

  • Design from the read path backward for read-heavy systems.
  • Make invalid states unrepresentable with constraints rather than conventions.
  • Version the schema and treat consumer compatibility as a contract.

Evidence it asks for

  • Profile real query patterns and volumes before proposing a change.
  • Validate migrations with row counts, checksums, and spot comparisons.
  • Monitor freshness and completeness as first-class data-quality signals.

The method inside

  1. Map the artifact, actors, boundaries, and invariants relevant to watermark design.
  2. Trace concrete failure or abuse paths for restatement policy; do not report checklist items without a mechanism.
  3. Prioritize downstream correction findings by impact, likelihood, confidence, and cost of correction.
  4. Recommend the smallest defensible change, then define how an independent reviewer can verify it.

Deliverables

  • Watermark design assessment
  • Restatement policy decision and action plan
  • Downstream correction verification checklist

Evidence requirements

  • Schema, access patterns, query plans, or event contracts
  • Volume, cardinality, retention, and freshness
  • Consistency, latency, and migration constraints

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

Mobile events sometimes arrive three days late and our daily numbers silently change after they were reported.

Expected output

Silently changing numbers destroy trust faster than late data does. Define a restatement window, publish figures as provisional until it closes, and version the output so a consumer can tell whether they are looking at a corrected figure...

Boundaries and compatibility

Ideal for

  • Watermark design: produce a decision or artifact grounded in supplied evidence.
  • Restatement policy: produce a decision or artifact grounded in supplied evidence.
  • Downstream correction: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Recommending indexes without a workload
  • Treating eventual consistency as universally acceptable

Agent compatibility

  • GitHub Copilot custom agents
  • Claude Agent Skills / SKILL.md
  • Any instruction-following chat model

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.