Failure handling

Failure handling & notifications

Jobs fail — a script errors, an API times out, a host is unreachable. Cronable gives you two layers of control: retries (try again automatically) and an error handler (run another job when one ultimately fails — most often to send a notification).

Retry a job automatically

Add a retry policy to any job. Cronable re-runs the job on failure, up to maxAttempts, waiting between attempts:

retry:
  maxAttempts: 3 # 1–20, required
  backoff: exponential # 'fixed' (default) | 'exponential'
  delayMs: 2000 # base delay between attempts (default 1000)
  • fixed waits delayMs between every attempt.
  • exponential grows the delay each attempt (roughly 2s, 4s, 8s …).

A run only counts as failed — and only triggers the error handler below — once every retry is exhausted.

Run a handler when a job fails (onFailure)

Point a job at another job to run as its error handler:

onFailure: alert-oncall # the id of another job

When the job ultimately fails, Cronable runs alert-oncall and passes the failed run to it as $parent, so the handler can report what failed and why:

ExpressionWhat it holds
{{ $parent.error }}the failure message (secrets masked)
{{ $parent.name }}the failed job’s name
{{ $parent.jobId }}the failed job’s id
{{ $parent.text }}the failed run’s captured output (if any)

Secret and credential values are masked in $parent.error before the handler ever sees them, so a notification can’t accidentally leak a token that appeared in an error message.

The handler is an ordinary job — give it an empty schedule so it only ever runs when it’s wired as a handler, never on its own:

schedule: []

Recipe: notify on failure

Wire an email, slack, telegram, or http job as the handler.

Email

# jobs/ops/alert-oncall.yaml — the handler
id: alert-oncall
name: Alert on-call
group: ops
type: email
to: oncall@example.com
subject: 'Job {{ $parent.name }} failed'
body: |
  "{{ $parent.name }}" ({{ $parent.jobId }}) failed.
 
  Error: {{ $parent.error }}
schedule: []
# jobs/reports/nightly-sync.yaml — the job that might fail
id: nightly-sync
name: Nightly sync
group: reports
type: terminal
command: /srv/scripts/sync.sh
onFailure: alert-oncall # wire the handler
schedule:
  - { kind: daily, time: '02:00' }

The email job sends via the provider configured under Settings → Email. If you omit to on the calling side, set a default sender there.

Slack

# jobs/ops/alert-slack.yaml — the handler
id: alert-slack
name: Alert Slack
group: ops
type: slack
credential: slack-bot # a slack credential (Settings → Secrets → Credentials)
channel: '#alerts'
message: ':rotating_light: {{ $parent.name }} failed — {{ $parent.error }}'
schedule: []

The slack and telegram job types post through a stored credential and handle message escaping for you, so {{ $parent.error }} drops straight into the text.

Any HTTP endpoint

For PagerDuty, Opsgenie, a Slack/Discord incoming webhook, or your own alerting service, use an http handler that POSTs to the endpoint. Pull the routing key or webhook URL from a secret ({{ $secrets.pagerduty_key }}) rather than hard-coding it.

It fires once per failure — safely

Cronable bounds the error-handler cascade so a misconfiguration can’t take down the machine. If handlers form a cycle (A.onFailure: B, B.onFailure: A) and both keep failing, the cycle is detected on the second hop and the re-dispatch is stopped with a logged warning — it never loops forever. A non-cyclic but excessively deep failing chain is likewise capped (at 25 handler hops). A normal single handler running once is unaffected.

Graph-wired equivalent

Instead of onFailure, you can add an afterParent schedule point to the handler with on: failed — the same effect, wired visually on the canvas. Use whichever reads better; they are equivalent.