SkillVaultskills Browse all 500 skills

Infrastructure · Version 1.8.2 · Reviewed 2026-08-02

Kubernetes Debugger

Review and harden CrashLoopBackOff and scheduling failures with evidence, explicit trade-offs, and a verification plan.

4 method steps 7 documented failure modes 5 diagnostic checks 7 quality gates

Turns events, pod states, and manifests into a prioritized Kubernetes investigation.

₹99 one-time

Get this skill archive

What this skill helps you do

  • CrashLoopBackOff
  • Scheduling failures
  • Networking diagnosis

How Kubernetes Debugger works

You provide

Pod events, manifests, probe and Service definitions

It inspects

Endpoint membership, exit codes, throttle counters

It decides

Ranked cause with the mechanism that explains every symptom

You verify

Re-run describe and confirm endpoints populate

What it checks first

Kubernetes Debugger turns events, pod states, and manifests into a prioritized Kubernetes investigation. Use it when the work involves CrashLoopBackOff, Scheduling failures, Networking diagnosis.

  1. Pod phase versus container state: `Running` with `ready: false` is a probe or dependency problem, not a scheduling problem.
  2. `kubectl get endpoints <svc>` before anything else when a Service returns 503 — an empty subset proves the selector or readiness gate is the fault.
  3. Last state `terminated` with exit code 137 (OOMKilled) or 143 (SIGTERM) separates memory pressure from graceful eviction.
  4. Events ordered by `lastTimestamp`, not the default sort, because the default hides the first occurrence that explains the cascade.
  5. Resource `requests` versus actual usage: scheduling failures follow requests, throttling follows limits.

Failure modes it recognizes

  • CrashLoopBackOff caused by a failing dependency at startup, where the exponential backoff makes the pod look idle rather than broken.
  • Readiness probe path differs from the Service `targetPort`, so endpoints never populate while the container is healthy.
  • CPU limits producing throttling that appears as random latency, visible only in `container_cpu_cfs_throttled_seconds_total`.
  • Liveness probe with a timeout shorter than the application pause, restarting a healthy but slow process and hiding the real cause.
  • PodDisruptionBudget blocking a node drain indefinitely because `minAvailable` equals the replica count.
  • Init container succeeding on retry, so the pod starts while the dependency it waited for is still unstable.
  • DNS resolution failures from `ndots:5` search-path expansion, causing latency spikes for external names.

Answers it will reject

  • Raising memory limits after an OOMKill without checking whether the working set grows unboundedly — this delays the same failure.
  • Using the same endpoint for liveness and readiness, which converts a transient dependency outage into a restart storm.
  • Deleting pods to "fix" a problem, destroying the evidence and resetting the backoff timer.
  • Blaming the CNI for connection failures before confirming the Service has endpoints and a NetworkPolicy is not denying traffic.

Decision rules it applies

  • If endpoints are empty, stop investigating the network and investigate readiness and label selectors.
  • If restart count is climbing but the container exits 0, treat it as a completed process rather than a crash: the command is wrong for a long-running workload.
  • Prefer increasing `initialDelaySeconds` or `failureThreshold` over disabling a probe; a disabled probe removes the signal permanently.
  • Scheduling failures are a requests-versus-capacity arithmetic problem — resolve them with numbers, not by restarting workloads.

Evidence it asks for

  • `kubectl describe pod <pod>` for the events tail and the last terminated state and reason.
  • `kubectl get events --sort-by=.lastTimestamp -A` to recover the first failure in a cascade.
  • `kubectl logs <pod> --previous` for the crashed container instead of the restarted one.
  • `kubectl get endpointslices -l kubernetes.io/service-name=<svc>` to prove or disprove Service membership.

The method inside

  1. Reconstruct the symptom timeline and define what healthy behavior would look like for CrashLoopBackOff.
  2. Rank hypotheses for scheduling failures by evidence, blast radius, and ability to explain every observed symptom.
  3. Run the cheapest discriminating check for networking diagnosis; update confidence only when evidence changes.
  4. Separate immediate stabilization, confirmed cause, contributing conditions, and prevention; finish with a reproducible verification.

Deliverables

  • CrashLoopBackOff assessment
  • Scheduling failures decision and action plan
  • Networking diagnosis verification checklist

Evidence requirements

  • Infrastructure code or configuration
  • Runtime topology and environment constraints
  • Plan, events, policies, and failure symptoms

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

My deployment shows all pods Running and healthy, but the Service returns 503 for every request.

Expected output

Check endpoint membership first: `kubectl get endpoints` will almost certainly show an empty subset. Running is not Ready, and a pod only joins the Service once its readiness probe passes and its labels match the selector. The probe path and the Service targetPort likely disagree...

Boundaries and compatibility

Ideal for

  • CrashLoopBackOff: produce a decision or artifact grounded in supplied evidence.
  • Scheduling failures: produce a decision or artifact grounded in supplied evidence.
  • Networking diagnosis: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Applying infrastructure changes without approval
  • Assuming cloud access or live resource visibility

Agent compatibility

  • GitHub Copilot custom agents
  • Claude Agent Skills / SKILL.md
  • Any instruction-following chat model

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.