﻿When a run fails, RunWisp shows it in the bell in the Web UI and the TUI. Add a
channel to also send it to Slack, Discord, Telegram, ntfy, Gotify, Pushover,
email, or a webhook.

```toml title="runwisp.toml"
[notifiers.slack-ops]
type        = "slack"
webhook_url = "${SLACK_OPS_URL}"

[tasks.backup-db]
cron   = "30 2 * * *"
run    = "/usr/local/bin/backup.sh"
notify = ["slack-ops"]
```

## The bell

Without any configuration, every run that counts as a failure adds a row to the
bell. By default that's a non-zero exit, a timeout, a run cut off by a crash, a
log that hit [`log_on_full = "kill"`](/configuration/tasks/#log_on_full), a [missed run](#missed-runs), and a service
that gave up restarting. The task's [`failures`](/configuration/tasks/#failures)
list decides what counts.

Bell rows are stored in the database, so the unread count is still right after
a restart. The bell needs no network. To turn it off, set
[`global_notifiers = []`](/configuration/notify/#global_notifiers).

## Adding a channel

Each channel is a `[notifiers.<id>]` block. The id is the name you use to refer
to it; the `type` picks the provider:

```toml title="runwisp.toml"
[notifiers.tg-oncall]
type      = "telegram"
bot_token = "${TG_BOT_TOKEN}"
chat_id   = "-1001234567890"
```

- **The id** must be unique. `inapp` is reserved for the bell, and the id can't
  contain `:`.
- **`type`** is one of `slack`, `discord`, `telegram`, `ntfy`, `gotify`,
  `pushover`, `smtp`, `sendmail`, or `webhook`. The other keys depend on the type; a key that belongs to another
  type is rejected.
- **`template_path`** works on every type: a file that replaces the built-in
  message template. Each provider page links its default template to start from.

The providers: [Slack](/notifications/providers/slack/),
[Discord](/notifications/providers/discord/),
[Telegram](/notifications/providers/telegram/),
[ntfy](/notifications/providers/ntfy/),
[Gotify](/notifications/providers/gotify/),
[Pushover](/notifications/providers/pushover/),
[Email (SMTP)](/notifications/providers/smtp/),
[Email (local MTA)](/notifications/providers/sendmail/), and
[Webhook](/notifications/providers/webhook/).

Declaring a channel doesn't send anything yet. You also say
[which failures go to it](#sending-alerts).

## Storing the secret

Every channel has a credential: a webhook URL, a bot token, or a password. Use
[`${...}` substitution](/configuration/substitution/) to keep it out of
`runwisp.toml`:

- **Environment variable:** `webhook_url = "${SLACK_OPS_URL}"`, with the value
  set in your shell, systemd unit, or container. The usual choice.
- **File:** `webhook_url = "${file:secrets/slack.url}"`. Relative paths are
  relative to `runwisp.toml`. Good when a secrets tool writes the value to disk;
  `chmod 600` the file.
- **Inline:** the value itself in `runwisp.toml`. Only for local testing: config
  files end up in git and in chat.

The value is read once, when the config loads. An unset variable or a missing
file stops the config from loading, with an error naming the key, so you find
out at startup instead of at the first failure.

If a delivery fails, the secret is replaced with `[redacted]` in the error
before it reaches the bell or the daemon log. Secrets are only ever sent to the
channel's own endpoint.

## Sending alerts

### For one task

Put [`notify`](/configuration/tasks/#notify) on the task or service. Its
failures go to those channels, plus everything in [`global_notifiers`](/configuration/notify/#global_notifiers) (the bell,
by default):

```toml title="runwisp.toml"
[services.payments-api]
run    = "/usr/local/bin/payments-api"
notify = ["slack-ops", "tg-oncall"]
```

### For every task

[`global_notifiers`](/configuration/notify/#global_notifiers) lists channels
that get every failure, from every task:

```toml title="runwisp.toml"
[notify]
global_notifiers = ["inapp", "slack-ops"]   # bell + Slack for everything
```

### For many tasks, or for other outcomes

`notify` only sends failures. To match tasks by name, or to send successes,
timeouts only, or other outcomes, write a [`[[route]]`](/configuration/routes/):

```toml title="runwisp.toml"
[[route]]
match     = { task = "deploy-*", kinds = ["succeeded"] }
notifiers = ["slack-ops"]
```

`notify`, `global_notifiers`, and routes all add up. If an event matches
several of them, each channel still gets one message.

### One channel, several destinations

Write `"<id>:<target>"` to reuse a channel's credentials with a different
target: a Slack channel or user, a Telegram chat id, or an email address for
`smtp` and `sendmail`:

```toml title="runwisp.toml"
[notifiers.slack]
type        = "slack"
webhook_url = "${SLACK_URL}"

[tasks.nightly-db-backup]
# ...
notify = ["slack:#ops"]

[tasks.weekly-deploy]
# ...
notify = ["slack:#deploys"]
```

Discord, Gotify, and webhook channels have no target to change, so they don't
accept this form. If many tasks use the same target, declare a separate notifier for
it instead.

## Missed runs

A scheduled run missed while the daemon was down
[counts as a failure](/concepts/scheduling/#missed-run-alerts) by default. It
goes to the same channels as the task's other failures, with nothing to set up.

For a task that's expected to miss runs, take `missed` out of its [`failures`](/configuration/tasks/#failures).
The missed run is still recorded; it doesn't alert:

```toml title="runwisp.toml"
[tasks.daytime-sync]
# ...
cron     = "*/30 9-17 * * 1-5"
notify   = ["slack-ops"]
failures = ["-missed"]
```

## Recovered runs

When a run succeeds and the task's run before it failed, a
[`recovered`](/configuration/routes/#matchkinds) notification goes to the same
channels as the task's failures, the bell included. A run that succeeds after
another success sends nothing.

## Testing a channel

Add a task that always fails, pointed at the channel:

```toml title="runwisp.toml"
[tasks.notify-test]
run    = "echo 'testing notifications' >&2; exit 1"
notify = ["slack-ops"]
```

Then reload the config and run it through the daemon:

```bash
runwisp reload
runwisp run --daemon notify-test
```

A message should arrive within a few seconds, and the bell gets a row. Use
`--daemon`: without a running daemon, `runwisp run` runs the task in the CLI and
sends no notifications. Repeats of the same failure within
[`coalesce_window`](/configuration/notify/#coalesce_window) are held back, so a
second test right after the first may not send a new message.

If only the bell shows the failure, open it and look for a
`notify.delivery_failed` warning with the error from the provider.

## Repeated failures

A task that fails every minute doesn't send a message every minute. Repeats of
the same failure update one bell row, and channels get the first failure, a
check-in every few repeats, and a summary at the end. With the defaults that's
about one message every ten minutes. A successful run ends the group, so a
failure after it alerts again right away. See
[`coalesce_window`](/configuration/notify/#coalesce_window) and
[`coalesce_limit`](/configuration/notify/#coalesce_limit).

## When delivery fails

Network errors, server errors (5xx), timeouts, and rate limits (429, honouring
`Retry-After`) are retried with backoff for up to
[`retry_budget`](/configuration/notify/#retry_budget). Other client errors
(like a 404 from a deleted webhook) fail at once.

When a delivery gives up, a `notify.delivery_failed` warning naming the task
goes to the bell. It only ever goes to the bell, and a `[[route]]` can't match
it. If you need a backup path, list a second channel in [`notify`](/configuration/tasks/#notify): both get
every event, so one keeps working when the other is down.
