Operations & scaling

Cronable is built to run continuously, for months, under heavy automation load — without degrading or filling the disk. Everything that grows over time (log files, run history, the database, the UI’s lists) is bounded: capped, pruned, or paginated. This page covers the knobs that keep a long-lived install healthy and what to back up.

The knobs below are environment variables read at daemon startup — and the retention knobs are also editable at runtime from Settings → Storage, no restart required (see Editing retention from the UI below). Sensible defaults mean most installs need to change nothing. See also Security and Updates.

Log files

Each run streams its (secret-masked) output to logs/<runId>.log. Two independent bounds keep logs/ from filling the disk:

• Per-run size cap. Once a run’s output crosses CRONABLE_LOG_MAX_MB, Cronable writes a one-line truncation marker and stops persisting further output to that file — while the run keeps executing and its live log keeps streaming to the UI. This bounds a runaway job (an infinite loop, a yes-style flood, a multi-GB export to stdout) so a single job can never fill the disk. The two knobs compose: the size cap bounds any one log; retention bounds how many accumulate.
• The UI never loads a giant log. The run view tails the log (it shows the most recent output and streams new lines live), so opening a run with a huge log stays fast — the page never pulls an unbounded file into the browser.

Run history & the database

Run rows, run output, and the audit/event trail live in data/cronable.db (SQLite). Run history grows with every execution, so four retention knobs bound it. They are off by default and are destructive (they DELETE rows irreversibly) — a fresh or existing install keeps everything until you opt in:

• In-flight runs are never pruned — only settled (succeeded / failed / cancelled / skipped) history is eligible, so a long-running job is never deleted out from under itself.
• Pruning reclaims log files too: deleting a run row also removes its logs/<runId>.log.
• Compaction is a separate opt-in. Pruning frees pages onto SQLite’s internal freelist but does not shrink the file on disk by itself. Set CRONABLE_VACUUM_ON_PRUNE=1 (default off) to have the prune pass run a VACUUM afterward — it rebuilds the file and returns the freed space to the OS. It’s off by default because VACUUM takes an exclusive, event-loop-blocking lock for the rewrite; enable it when reclaiming disk matters more than a brief pause, and only alongside retention (with retention off nothing is pruned, so there’s nothing to reclaim).
• The Executions view paginates. The run list loads a page at a time (“Load more”) rather than the whole history, so it stays responsive no matter how many runs have accumulated.

A good starting point for a busy install: enable a count cap (e.g. CRONABLE_RUN_COUNT_CAP=200) so each job keeps a useful window of recent history without unbounded growth, and/or an age window that matches how far back you actually look.

Trigger dedup state

Polling triggers (folder-watch, inbox, calendar-watch, telegram-watch, sheet-watch, hostaway-watch) keep a small per-job “seen” set so they fire once per item, never twice. That set grows as new items arrive, so it has its own bound:

Unlike the run/event retention knobs above, this one is on by default (non-zero) — the seen-sets aren’t user-visible history, so keeping them bounded needs no opt-in. The prune only trims old dedup marks; it never evicts a trigger’s cursor (the “where I last read to” position), so no event is ever re-fired or missed. Set it to 0 to disable the cap.

Concurrency

When more jobs are due at once than the ceiling allows, the extra runs wait (or are skipped per the job’s overlap policy) rather than overwhelming the machine — so a burst of due jobs can’t spike load unboundedly.

Triggering a job on demand

Most jobs run on a schedule or chain off a parent, but you can also fire one right now — from the dashboard’s “Run now”, or headlessly with the CLI:

cronable run nightly-backup               # trigger and return (prints the run id)
cronable run deploy --wait                 # block until it finishes; exit 0 only if the run SUCCEEDED
cronable run deploy --wait && notify ok     # gate the next step on a successful run

cronable run <jobId> triggers the job through the daemon (the same path as “Run now”). With --wait it polls the run to completion and exits non-zero if the run didn’t succeed, so a CI step or a wrapper script can trigger a Cronable job and react to its outcome. --timeout (default 300s) bounds the wait — a timeout exits non-zero, but the run keeps executing on the daemon. An unknown job id or a job that can’t start (an afterParent step whose parent hasn’t run, or one already running) fails with a clear message.

Monitoring

Five endpoints let you watch a running install. All are on the daemon, at the same host/port as the API (127.0.0.1:8787 by default).

/api/health and /api/ready are unauthenticated so a container HEALTHCHECK (and depends_on: service_healthy) can reach them without a credential — /api/ready is the one to gate orchestration on (see Troubleshooting → Health checks).

/api/version is likewise unauthenticated — it returns only the build version string and an integer API version (no state, paths, or secrets), so a hosted UI can check compatibility before it logs in, and you can confirm which build is deployed (e.g. after a self-update).

/api/metrics is authenticated (run counts aren’t for anonymous callers), so a scraper sends the CRONABLE_TOKEN bearer:

curl -s -H "Authorization: Bearer $CRONABLE_TOKEN" http://127.0.0.1:8787/api/metrics
# → { "uptimeSeconds": 3600,
#     "runs": { "total": 1284, "active": 2, "byState": { "succeeded": 1200, "failed": 80, "running": 2, … } },
#     "license": "valid", "gitSync": "idle", "configErrors": 0 }

It returns aggregate counts + coarse health enums — process uptimeSeconds; runs with the total, the active count (starting + running + waiting), and a byState breakdown; and three daemon-health signals: license (valid | grace | blocked | unlicensed), gitSync (idle | syncing | conflict | error | disabled), and configErrors (a count of job files that failed to load). The run counts are a single bounded GROUP BY (one row per run state) and the health reads are all in-memory (the license check is one sub-millisecond in-process token verify), so it stays cheap to poll no matter how much history has accumulated, and it never exposes run output, job content, or secret values — only counts and coarse state.

Scraping with Prometheus

Add ?format=prometheus to the same endpoint and Cronable returns the identical numbers as Prometheus text exposition (format 0.0.4, content-type: text/plain; version=0.0.4) instead of JSON — so a Prometheus/OpenMetrics scraper can ingest it natively, no exporter sidecar:

curl -s -H "Authorization: Bearer $CRONABLE_TOKEN" \
  "http://127.0.0.1:8787/api/metrics?format=prometheus"
# HELP cronable_uptime_seconds Daemon process uptime in seconds.
# TYPE cronable_uptime_seconds gauge
cronable_uptime_seconds 3600
# HELP cronable_runs Runs recorded in the database, labeled by state (current count, not a running total).
# TYPE cronable_runs gauge
cronable_runs{state="succeeded"} 1200
cronable_runs{state="failed"} 80
cronable_runs{state="running"} 2
# HELP cronable_runs_active Runs currently in flight (starting|running|waiting).
# TYPE cronable_runs_active gauge
cronable_runs_active 2
# HELP cronable_license_state Commercial license state (1 for the current state, else 0).
# TYPE cronable_license_state gauge
cronable_license_state{state="valid"} 1
cronable_license_state{state="grace"} 0
cronable_license_state{state="blocked"} 0
cronable_license_state{state="unlicensed"} 0
# HELP cronable_git_sync_state Jobs-repo git-sync state (1 for the current state, else 0).
# TYPE cronable_git_sync_state gauge
cronable_git_sync_state{state="idle"} 1
cronable_git_sync_state{state="syncing"} 0
cronable_git_sync_state{state="conflict"} 0
cronable_git_sync_state{state="error"} 0
cronable_git_sync_state{state="disabled"} 0
# HELP cronable_config_errors Job files that failed to load and will not run (0 = all valid).
# TYPE cronable_config_errors gauge
cronable_config_errors 0

Everything is a gauge. cronable_runs{state=…} is the current count per state (history pruning can lower it), not a monotonic counter. The three daemon-health metrics — cronable_license_state{state=…}, cronable_git_sync_state{state=…}, and cronable_config_errors — surface degraded conditions the run counts can’t: a blocked license (jobs stop running), a stuck jobs-repo git sync, or job YAML that failed to load. Each state-labeled health gauge emits every state as a present 0/1 (exactly one is 1), so an alert rule can reference a specific state without worrying about an absent series. Any other or absent format value returns the JSON above (now including the license, gitSync, and configErrors fields), so existing JSON scrapers keep working unchanged.

The endpoint stays behind the bearer, so point the scrape config’s authorization at your CRONABLE_TOKEN and keep the target on loopback (or your private network) — never expose port 8787:

# prometheus.yml
scrape_configs:
  - job_name: cronable
    metrics_path: /api/metrics
    params:
      format: [prometheus]
    authorization:
      type: Bearer
      credentials: '<CRONABLE_TOKEN>' # or credentials_file: /etc/prometheus/cronable.token
    static_configs:
      - targets: ['127.0.0.1:8787']

Then chart cronable_runs_active for in-flight load and alert on a rising cronable_runs{state="failed"}.

Alerting on daemon health

The health gauges are built for alerting on a daemon that’s still up (serving 200s) but degraded — the failures a plain uptime check misses. Example rules:

# cronable.rules.yml
groups:
  - name: cronable
    rules:
      - alert: CronableLicenseBlocked
        # Blocked → the daemon stops running jobs. Page on this.
        expr: cronable_license_state{state="blocked"} == 1
        for: 5m
        labels: { severity: critical }
        annotations: { summary: 'Cronable license is blocked — jobs are not running' }
 
      - alert: CronableLicenseGrace
        # A softer warning window before it blocks — renew now.
        expr: cronable_license_state{state="grace"} == 1
        for: 1h
        labels: { severity: warning }
        annotations: { summary: 'Cronable license in grace — renew before it blocks' }
 
      - alert: CronableGitSyncStuck
        # conflict = a pull that won't rebase; error = a failing fetch/push. Either means the jobs
        # repo has stopped converging and UI edits may not be syncing.
        expr: cronable_git_sync_state{state="conflict"} == 1 or cronable_git_sync_state{state="error"} == 1
        for: 15m
        labels: { severity: warning }
        annotations: { summary: 'Cronable jobs-repo git sync is stuck ({{ $labels.state }})' }
 
      - alert: CronableConfigErrors
        # Job files the daemon couldn't load — those jobs silently never fire until fixed.
        expr: cronable_config_errors > 0
        for: 10m
        labels: { severity: warning }
        annotations: { summary: '{{ $value }} Cronable job file(s) failed to load' }

Because each state-labeled gauge always publishes every state as a 0/1, these expressions never break on a missing series — cronable_license_state{state="blocked"} reads 0 until it is genuinely blocked, so the alert simply stays inactive rather than erroring on an absent metric.

Storage & retention at a glance

GET /api/storage (bearer) answers “how much is this install storing, and what bounds it?” in one call — so you don’t have to read env vars or open the database. It returns the current usage and the retention config side by side:

curl -s -H "Authorization: Bearer $CRONABLE_TOKEN" http://127.0.0.1:8787/api/storage
# → { "retention": { "logRetentionDays": 30, "logMaxBytes": 104857600, "runRetentionDays": 0,
#                     "runCountCap": 0, "eventRetentionDays": 0, "eventCountCap": 0,
#                     "triggerStateCap": 10000, "vacuumOnPrune": false },
#     "usage": { "dbBytes": 5242880, "logStorageBytes": 18874368, "runCount": 1284,
#                "eventCount": 640, "triggerStateCount": 312 } }

• usage — dbBytes is the on-disk size of cronable.db; logStorageBytes is the total tracked size of per-run log files; and the three counts are the current rows in runs, events, and trigger_state. It’s the same usage the Settings → Storage panel shows in the UI.
• retention — the effective retention config: the environment defaults above with any Settings → Storage overrides layered on (a stored override wins; an unset knob falls back to the env default). So it always reflects what the daily prune will actually do, whether a knob was set from an env var or from the UI.

It’s bounded by construction — one stat of the database file plus three cheap COUNT/SUM queries, never a filesystem walk — so it stays cheap to poll no matter how much history has accumulated. (logStorageBytes therefore slightly over-counts a log whose file was age-pruned while its run row was kept — fine as a usage indicator.)

Editing retention from the UI

The retention knobs aren’t environment-only — an operator can set them at runtime under Settings → Storage (Advanced mode), with no restart. The panel shows current usage plus an editable field per knob; Save writes your overrides and the next daily prune picks them up.

How overrides resolve, per knob and independently:

• A value you set in the UI wins over the matching CRONABLE_* env default.
• A knob you leave untouched keeps deferring to its env default — the UI sends only the fields you actually change, so an install with no overrides behaves exactly as the env-only defaults above.
• 0 means “off” and is a real value: it wins over a non-zero env default (so you can turn a knob off from the UI even if the environment enabled it). Setting 0 never deletes more — it stops pruning that dimension.

Because these prunes irreversibly delete runs, events, and their log files, the panel warns and asks you to confirm before you tighten a limit (lowering a window, or switching a knob from off to on). Overrides persist in .cronable/settings.json under retention and are reported back by GET /api/storage as the effective config shown above.

CRONABLE_LOG_MAX_MB is not editable here. It is a write-time per-run log size cap enforced as the log is written — not part of the daily prune pass — so it stays environment-only.

What to back up

data/ and .cronable/ are daemon-owned and hold everything stateful:

• data/cronable.db — run history plus the encrypted secrets and credentials.
• .cronable/ — global variables and the graph layout.
• The encryption key (CRONABLE_ENCRYPTION_KEY) — back this up separately from the database. It is never stored in data/, and without it the encrypted secrets are unrecoverable by design. See Secrets & credentials.

Your job definitions live in the separate jobs repository (their own git history), so they are backed up by that repo — not by the above.

Taking a consistent backup

data/cronable.db is SQLite, so take a consistent snapshot rather than copying the file while the daemon is mid-write. The safe hot-copy is SQLite’s own online backup — no downtime:

sqlite3 data/cronable.db ".backup 'backups/cronable-$(date -u +%Y%m%d).db'"
cp -r .cronable "backups/cronable-config-$(date -u +%Y%m%d)"   # variables + layout

Store those alongside a copy of CRONABLE_ENCRYPTION_KEY kept separately (a password manager or sealed secret store) — the database is useless for secrets without it. A nightly cronable job of type terminal running the snapshot is the simplest way to automate it.

Restoring

1. Stop the daemon (systemctl stop cronable, or Ctrl-C in dev) so nothing is writing.
2. Put the snapshot back as data/cronable.db and restore .cronable/.
3. Ensure the same CRONABLE_ENCRYPTION_KEY is in the environment — a different key boots but fails loudly the first time it touches a secret (there is no recovery without the original key).
4. Restart the daemon. It reconciles the jobs directory on boot, so pull the jobs repo first if you are restoring onto a fresh machine.

Export & import jobs

Move a job — or a whole group — between installs as a portable bundle, independent of the jobs git repo. Two auth-gated endpoints:

• GET /api/jobs/export returns a bundle of your raw job definitions (optionally one group):

  curl -s -H "Authorization: Bearer $CRONABLE_TOKEN" http://127.0.0.1:8787/api/jobs/export
  # → { "version": 1, "exportedAt": "…", "jobs": [ … ] }
  # one group only:
  curl -s -H "Authorization: Bearer $CRONABLE_TOKEN" \
    'http://127.0.0.1:8787/api/jobs/export?group=backups'

  The bundle is secret-safe: jobs reference secrets only as {{ $secrets.NAME }}, never a value — so it’s safe to share or check in. On the destination install, create the referenced secrets + credentials by name.

• POST /api/jobs/import takes such a bundle and is create-only — it creates each job that doesn’t already exist and never overwrites one that does:

  curl -s -X POST -H "Authorization: Bearer $CRONABLE_TOKEN" -H 'content-type: application/json' \
    --data @bundle.json http://127.0.0.1:8787/api/jobs/import
  # → { "imported": ["backup-nightly"], "skipped": ["heartbeat"], "errors": [] }

  imported = created; skipped = an id that already existed (left untouched); errors = a job that failed validation (one bad job never aborts the rest). Imported jobs go through the normal write → reconcile → audit path, so they appear on the canvas and begin scheduling immediately.

  Preview first with ?dryRun=1. Before committing a bundle, add the flag to see exactly what would happen — every job is parsed and conflict-checked the same way, but nothing is written (no file, no schedule, no audit):

  curl -s -X POST -H "Authorization: Bearer $CRONABLE_TOKEN" -H 'content-type: application/json' \
    --data @bundle.json 'http://127.0.0.1:8787/api/jobs/import?dryRun=1'
  # → same { "imported": […], "skipped": […], "errors": […] } shape,
  #   where "imported" now means "would be created". Nothing lands on disk.

  Use it to vet a bundle from elsewhere — confirm which jobs are new, which collide with ids you already have, and which fail validation — then re-run without the flag to actually import.

  From the CLI, cronable export and cronable import wrap these endpoints — no curl or bearer token needed. cronable export --out backup.json (add --group ops, or omit --out to pipe to stdout); cronable import backup.json (create-only, --dry-run to preview, - or a pipe for stdin). cronable import exits non-zero if any job fails to import, so a backup/restore script can safely gate on it.