cron-watchdog-debug

Cron Watchdog Debug

Use this skill for cron wake, watchdog, scheduled OpenClaw job, and restore-oracle incidents.

Non-Negotiables

Before proposing a fix, produce:

1. Deployment-state proof.
2. Watchdog path diagram with every edge marked unknown, verified-good, or verified-bad.
3. Hypothesis table with fastest falsifier and status.
4. Cron Watchdog Handoff.

Do not treat "cron route ran" as proof that an OpenClaw scheduled job ran. Keep these states separate: Vercel Cron invoked, watchdog authorized, persisted wake due, sandbox woke, AI Gateway token refreshed, OpenClaw cron scheduler loaded jobs, user-visible delivery happened.

Evidence First

For live deployments, create one artifact root:

RUN_TS="$(date -u +%Y%m%dT%H%M%SZ)"
ART=".agent-runs/cron-debug/$RUN_TS"
mkdir -p "$ART"/{admin,vercel,sandbox,workflow}

Collect before editing code:

• .vercel/project.json versus the intended project/team/deployment.
• git rev-parse HEAD and git ls-remote origin main.
• Live GET /api/status.
• Live GET /api/admin/sandbox-diag.
• Live GET /api/admin/logs filtered for watchdog., sandbox., gateway., cron, and restore.
• Live GET /api/admin/launch-verify if the incident involves launch readiness.
• Live GET or POST /api/cron/watchdog only with valid cron authorization.

If a release env file is supplied, source it only in the shell running live commands. Never print or save raw secrets.

Watchdog Path Diagram

Use this shape and mark each edge:

Vercel Cron tick -> /api/cron/watchdog auth -> runSandboxWatchdog
  -> deployment contract -> metadata read -> busy/stale/running/stopped branch
  -> cron wake key read -> ensureSandboxReady? -> token refresh
  -> OpenClaw cron jobs present in sandbox -> scheduled job executes -> delivery visible

For a running sandbox due soon, include the parallel branch:

running sandbox -> gateway probe -> cronNextWakeMs <= now + 60s
  -> force AI Gateway token refresh -> restore oracle cycle -> no duplicate wake

Runtime Probes

Read admin surfaces first. Then, only if needed, inspect the sandbox with npx sandbox or the app admin SSH/exec fallback.

Read-only sandbox checks:

set -eu
echo "== process =="
ps -eo pid,ppid,comm,args | grep -E "[o]penclaw|[n]ode" || true
echo "== ports =="
(ss -ltnp || netstat -ltnp) 2>/dev/null | grep -E ":(3000|8787)\\b" || true
echo "== cron files =="
ls -la /home/vercel-sandbox/.openclaw/cron 2>/dev/null || true
echo "== cron jobs shape =="
node - <<'NODE'
const fs = require('fs');
for (const p of [
  '/home/vercel-sandbox/.openclaw/cron/jobs.json',
  '/home/vercel-sandbox/.openclaw/cron/jobs-state.json',
]) {
  try {
    const j = JSON.parse(fs.readFileSync(p, 'utf8'));
    console.log(p, JSON.stringify({
      version: j.version ?? null,
      jobCount: Array.isArray(j.jobs) ? j.jobs.length : Object.keys(j.jobs || {}).length,
      jobIds: Array.isArray(j.jobs) ? j.jobs.map((x) => x.id) : Object.keys(j.jobs || {}),
    }));
  } catch (err) {
    console.log(p, 'unreadable', err.message);
  }
}
NODE

Do not print job payload text if it may contain user data. Prefer shape, counts, ids, and next-run timestamps.

Store Evidence

Cron wake depends on store keys written through the lifecycle layer. Do not hardcode Redis prefixes in app code. For debugging, use app/admin surfaces when available; if direct store inspection is unavoidable, record key names and redacted shapes only.

High-signal facts:

• cronNextWakeMs exists and is due, future, missing, or malformed.
• cronJobsJson structured record exists, has a SHA-256, job count, and source.
• lastRestoreMetrics.cronRestoreOutcome is present or missing after wake.
• watchdog report contains cron.wake, token.refresh, probe, and restore.prepare checks.

Hypotheses To Compare

Keep all rows visible as they are ruled out:

Hypothesis	Evidence for	Evidence against	Fastest falsifier	Status
Vercel Cron did not invoke the route			Vercel logs for /api/cron/watchdog
Cron auth rejected the request			route response/log status 401 vs report
Wake key was never persisted			store/admin evidence for cronNextWakeMs
Wake key is future or stale			compare cronNextWakeMs to current UTC
Watchdog intentionally skipped idle sandbox			cron.wake check message and metadata status
Sandbox woke but token refresh failed			token.refresh check and AI Gateway token metadata
Jobs file missing or malformed in sandbox			sanitized cron file shape from sandbox
OpenClaw scheduler loaded jobs but delivery failed			OpenClaw logs plus channel lastForward/user-visible evidence
Restore oracle or hot spare side effect changed cron state			restore.prepare check and restore metrics

Safe Fix Boundaries

• src/app/api/cron/watchdog/route.ts owns cron-route auth and response shape.
• src/server/watchdog/run.ts owns watchdog decision order and report checks.
• src/shared/watchdog.ts owns report types.
• src/server/sandbox/lifecycle.ts owns cron persistence keys, stop/touch persistence, and wake behavior.
• src/server/sandbox/cron-persistence.test.ts and src/server/watchdog/run.test.ts own regression coverage.
• lat.md/sandbox-lifecycle.md owns the architecture narrative for cron wake.

Do not edit channel routes while debugging cron unless evidence proves the scheduled job reached channel delivery and failed there. Switch to the relevant channel skill for that phase.

Verification

Pick the smallest verification that covers the changed edge:

node scripts/verify.mjs --steps=test,typecheck
node --test src/server/watchdog/run.test.ts
node --test src/server/sandbox/cron-persistence.test.ts
lat check

For live fixes, include before/after evidence from /api/cron/watchdog, /api/admin/logs, and sandbox cron file shape.

Handoff

Use references/handoff-template.md for incident reports.