﻿A task can try again when it fails, and can be stopped when it runs too long.
Both are set per task:

```toml title="runwisp.toml"
[tasks.publish-feed]
cron           = "*/15 * * * *"
run            = "/usr/local/bin/publish.sh"
retry_attempts = 3
retry_delay    = "30s"
retry_backoff  = "exponential"
timeout        = "10m"
```

## What counts as a failure

By default, a run fails when the command exits non-zero, times out, is stopped
by [`log_on_full = "kill"`](/configuration/tasks/#log_on_full), or was still running when the daemon crashed. A
scheduled run [missed](/concepts/scheduling/#missed-run-alerts) while the daemon
was down counts too. A failed run turns red in the UI, sends
[notifications](/notifications/), and can be retried.

You can change this per task with [`failures`](/configuration/tasks/#failures),
for example to treat only some exit codes as failures. The full rules are in
[What counts as a failure](/configuration/tasks/#what-counts-as-a-failure).

## Retrying failed runs

[`retry_attempts`](/configuration/tasks/#retry_attempts) is the number of extra
tries after the first one fails. It's `0` by default, so nothing retries until
you set it. The retry keys are per task; they can't go in
[`[defaults]`](/configuration/defaults/).

A run is retried only when it failed **and** the task's [`failures`](/configuration/tasks/#failures) list counts
it as a failure. With `failures = ["1-23"]`, exit code `42` isn't a failure, so
it isn't retried either.

Some endings never retry:

- `stopped`: someone stopped the run, or [`on_overlap = "kill"`](/configuration/tasks/#on_overlap) replaced it.
  This also ends the retry chain.
- `skipped`: the previous run is still going.
- `missed`: nothing ran, so there is nothing to retry.

Adding `stopped` or `missed` to `failures` doesn't change this.

A retry uses exactly the same [parameter](/concepts/parameters/) values as the
run it retries.

### Waiting between attempts

[`retry_delay`](/configuration/tasks/#retry_delay) is the wait before the first
retry. [`retry_backoff`](/configuration/tasks/#retry_backoff) decides whether
later waits stay the same or grow:

```toml title="runwisp.toml"
[tasks.flaky-fetch]
# ...
retry_attempts = 4
retry_delay    = "10s"
retry_backoff  = "exponential"   # waits 10s, 20s, 40s, 80s
```

No wait is ever longer than 5 minutes, whatever the backoff.

Stopping the task while a retry is waiting cancels that retry and ends the
chain. So does a reload that removes the task.

### Retries in the history

Every attempt is its own run, with its own exit code and log. The history
numbers them with `retry_attempt`: `0` for the first try, `1` for the first
retry, and so on. If an attempt fails and a later one succeeds, you can still
read what the failed one printed.

Each failed attempt is a failure event of its own. Repeats of the same failure
are grouped by [`coalesce_window`](/configuration/notify/#coalesce_window), so a
retry chain usually sends one message, not one per attempt.

## Timeout

[`timeout`](/configuration/tasks/#timeout) is the longest one attempt may run:

```toml title="runwisp.toml"
[tasks.heavy-job]
cron    = "0 3 * * *"
run     = "/usr/local/bin/heavy-job.sh"
timeout = "30m"
```

When the time is up, RunWisp sends the task's
[`stop_signal`](/configuration/tasks/#stop_signal) (`SIGTERM` by default) to the
whole process group, waits [`graceful_stop`](/configuration/tasks/#graceful_stop)
(`5s` by default), then kills whatever is left. The run is recorded as
`timeout`. Manual stops, [`on_overlap = "kill"`](/configuration/tasks/#on_overlap), and daemon shutdown stop runs
the same way.

- The limit is per attempt. Each retry gets the full time again, and the wait
  between attempts doesn't count.
- Without a `timeout` on the task or in [`[defaults]`](/configuration/defaults/),
  runs have no time limit. Set `timeout = "0s"` on a task to turn off a
  `[defaults]` timeout.

## Services restart instead

Services don't retry; they restart. By default a service is started again after
every exit, with a growing delay between restarts. If it keeps failing before it
has been up for [`healthy_after`](/configuration/services/#healthy_after),
RunWisp stops trying and marks it FATAL. The keys are
[`restart`](/configuration/services/#restart),
[`restart_delay`](/configuration/services/#restart_delay),
[`restart_backoff`](/configuration/services/#restart_backoff), and
[`restart_attempts`](/configuration/services/#restart_attempts); see
[When a service can't start](/configuration/services/#when-a-service-cant-start-fatal).

If you want a limited number of tries, use a task with [`retry_attempts`](/configuration/tasks/#retry_attempts). If you
want a process kept alive indefinitely, use a service.
