The check can't be the agent

Whatever tells you an agent failed can't be the agent. If the alert, the report or the retry depends on the same model, the same account and the same budget as the work, it fails along with the work and stays silent about it. The fix isn't a smarter agent. The fix is a small, unremarkable piece of plain code that sits outside the agent and can't fail for the same reason.

Here are two cases. The first is someone else's and the second is ours, and they break in opposite directions.

Case one: 19 hours of nothing

On 21 September 2026 a team described this on r/AI_Agents, and the thread drew 20 comments. Their account hit its weekly usage limit. For about 19 hours after that, every scheduled job that starts a model died on launch. The plain scripts on the same schedule kept running as if nothing had happened. The agent that would have reported the failure was one of the jobs that never started.

Nothing was wrong with any single job. They shared one blast radius: one account, one cap, and one monitor inside the thing being monitored.

Case two: green, with zero of five steps done

Kindling is our product that helps specialists turn their own reactions into posts. It ran a background job made of five steps. Each step caught its own error, and then the job returned success. The queue recorded the job as done, so the retry logic never ran: from where it stood, there was nothing to retry. The report showed green. The job had completed zero of its five steps.

This one didn't go silent. It reported something false. The cause is the same, though: the code doing the work also decided whether the work had happened, and it marked itself as passing.

An outside write-up on monitoring agents in production makes the same point at the infrastructure level. Agents exit cleanly, with no errors in the logs, while quietly failing at the thing they were supposed to do. CPU, memory and exit codes look healthy, and the result is still missing.

The rule

Our internal production pipeline runs on the same rule in different words: a process that has hung can't be the one that reports hangs. The count of stuck work comes from a reader at the entry point, and that reader keeps running after a worker dies.

We think that's the whole idea. The rest of this piece applies it four ways. The code below is a sketch of the pattern, not a copy of anyone's system.

Fix 1: a plain-code gate before the model starts

The cheapest way to protect a quota is to avoid spending it. Before a scheduled job starts a model, plain code checks whether there's any work at all. On a quiet day the gate costs nothing and the shared cap stays untouched.

# runs on the schedule; no model involved
pending = db.count("select 1 from inbox where handled = false")
if pending == 0:
    log("nothing to do")
    exit(0)          # quiet day: zero quota spent
run_agent(batch=pending)

This fix came up in the thread. In our view it's also the one that most often goes missing in inherited builds, because "start the agent and let it decide" feels simpler.

Fix 2: "limit hit" and "login expired" are different failures

A quota hit clears on its own. An expired credential needs a person. If both show up as a generic error, the retry loop runs into the cap again and again, and nobody hears about the login.

try:
    result = run_agent(job)
except ProviderError as e:
    if is_quota_message(e):        # match the provider's literal limit text
        mark(job, "deferred:quota")  # no retry storm; resume after reset
    elif is_auth_error(e):
        mark(job, "blocked:auth")    # page a human, do not retry
        notify_outside_channel(job, e)
    else:
        raise

Match the provider's literal limit message, so a quota hit is labeled as quota and nothing else. Kindling's fix used the same idea for a different pair of states: a missing key now means "not done", not "failure". "Didn't happen yet" and "broke" call for different responses, so they get different labels.

Fix 3: must-run jobs get their own budget

If a nightly batch job shares a key with the morning digest your clients read, the batch job can use up the digest's budget. Keeping budgets separate, as the thread also suggested, turns one big blast radius into several small ones.

# budgets.yml
must_run:
  key: AGENT_KEY_CRITICAL
  jobs: [client_digest, billing_sync, failure_report]
best_effort:
  key: AGENT_KEY_BATCH
  jobs: [backfill, enrichment, experiments]

The failure report sits in must_run on purpose. If it shares a budget with the work it reports on, case one happens again.

Fix 4: a heartbeat the agent can't fake

This is the core of it. Each job writes a timestamp when it really finishes. A separate cron job with no model in it reads those timestamps and raises an alert when one is too old. The watcher doesn't need to understand the work. It only needs to keep running when the work stops.

# job side, last line of a successful run
write_file(f"/var/heartbeat/{job}.ts", now())

# watcher side: plain cron, no model, separate key
for job, max_age in EXPECTED.items():
    age = now() - read_ts(f"/var/heartbeat/{job}.ts")
    if age > max_age:
        send_sms(f"{job} silent for {age}")

A heartbeat only proves something if it's written when the work really succeeded. That leads back to Kindling. Its fix makes the job fail with a list of the steps that failed, so the queue finally sees a failure and the retries run. We also narrowed the lock to the one step that isn't safe to repeat, so retrying the job doesn't redo work that already went through.

failed = []
for step in steps:
    try:
        step.run()
    except Exception as e:
        failed.append((step.name, e))

if failed:
    raise JobFailed(failed)   # the queue sees it; retries run
return ok()

# lock held only around the single non-idempotent step

What we'd check first

If you run scheduled agent jobs, ask these three questions before anything else. They're our list, not a standard.

• If your account hit its limit tonight, what would tell you, and does that thing run on the same account?
• Which jobs share a key, and which of them are you contractually or reputationally obliged to run?
• Can any job report success without having produced its output?

If any answer is "I'm not sure", that's the job to fix first.

FAQ

Isn't this just normal monitoring? Mostly, yes. What's different with agents is that the usual signals look healthy while the result is missing. A clean exit, a done flag in the queue and a green report were all present in the cases above. Check the output, not the process.

Can't I have a second agent watch the first one? You can, as long as it doesn't share an account, a budget or a failure mode with the first. If one usage cap can stop both, it's the same agent with two names. In our experience, a plain cron job that reads a timestamp is cheaper and harder to break.

How do I stop retries from making a quota problem worse? Treat a quota hit as "deferred until reset", not as an error to retry. Match the provider's limit message explicitly, and keep it separate from authentication failures, which need a person.

Doesn't catching every error make a job more robust? Only if the caught errors are reported. Kindling's job caught every step's error and returned success, which is how it showed green with zero of five steps done. Catch errors so you can collect them, then fail with the list.

What's the smallest version of this worth shipping this week? In our view, a heartbeat file per must-run job and a model-free cron that alerts when one goes stale. It's a small change, and it covers the 19-hour case.

A related case

The Kindling fix above was a small part of building a product that has to stay correct while nobody's watching: background jobs, paid data sources and a feed that mustn't get quietly shorter. The full case is at iloblique.com/kindling.

If you have something worth building, we'd like to hear about it.

Sources

• r/AI_Agents thread, 21 September 2026, 20 comments: https://www.reddit.com/r/AI_Agents/comments/1wm50se/all_our_scheduled_agent_jobs_died_for_19_hours_on/
• Kindling: https://kindling.iloblique.com