﻿A health check is a `curl` on a schedule. The settings around it decide how
soon you hear about an outage and how much noise a flaky service makes.

To check a service RunWisp runs itself, and restart it when the check keeps
failing, use its [`health_check`](/configuration/services/#health_check)
instead.

## The task

```toml title="runwisp.toml"
[tasks.healthcheck-api]
group          = "Health checks"
description    = "External probe of the public API every 5 minutes"
cron           = "*/5 * * * *"
on_overlap     = "skip"
catch_up       = 0
timeout        = "30s"
retry_attempts = 2
retry_delay    = "30s"
retry_backoff  = "linear"
keep_runs      = 500
notify         = ["slack-ops"]

run = """
curl --silent --show-error --fail-with-body \\
     --max-time 25 --connect-timeout 5 \\
     --user-agent 'runwisp-healthcheck/1.0' \\
     https://api.example.com/health
"""
```

## What each setting does

- **[`cron = "*/5 * * * *"`](/configuration/tasks/#cron):** you learn about an outage within five minutes. Use
  `* * * * *` for tighter checks or `*/15 * * * *` for internal tools.
- **[`on_overlap = "skip"`](/configuration/tasks/#on_overlap):** a slow probe doesn't get a second one stacked on
  it. The dropped firing is recorded as `skipped`, which isn't a failure and
  doesn't alert.
- **[`catch_up = 0`](/configuration/tasks/#catch_up):** probes missed while the daemon was down are not run
  when it comes back. They would only report on a past you can't change. See
  [`catch_up`](/configuration/tasks/#catch_up).
- **[`timeout = "30s"`](/configuration/tasks/#timeout) with `curl --max-time 25`:** curl gives up first, so the
  log shows curl's own error message. The [`timeout`](/configuration/tasks/#timeout) is the backstop.
- **[`retry_attempts = 2`](/configuration/tasks/#retry_attempts), [`retry_delay = "30s"`](/configuration/tasks/#retry_delay), [`retry_backoff = "linear"`](/configuration/tasks/#retry_backoff):**
  a failed probe is tried again after 30s, then after 60s more. A short blip
  heals on its own and the task goes green again within about 90 seconds.
  Each failed attempt still alerts, but repeats are grouped (see below). See
  [`retry_attempts`](/configuration/tasks/#retry_attempts).
- **[`keep_runs = 500`](/configuration/tasks/#keep_runs):** five-minute checks make 288 runs a day, so this keeps
  well over a day of history. See [`keep_runs`](/configuration/tasks/#keep_runs).
- **[`notify`](/configuration/tasks/#notify):** alerts `slack-ops` on every failed run. During a long outage,
  repeats inside [`coalesce_window`](/configuration/notify/#coalesce_window)
  (default `1h`) are grouped: the first alert goes out right away, then
  occasional check-ins with a count, instead of one message every five minutes.
  When the check passes again, a [`recovered`](/configuration/routes/#matchkinds)
  alert says so.
- **`--fail-with-body`:** makes curl exit non-zero on an HTTP 4xx or 5xx and
  print the response body into the log. Without it, a `500` counts as success.
  It needs curl 7.76 or newer; on older curl use `--fail`.

## Bell only, no messages

For a flaky service where you'd rather look than be messaged, drop the retries
and [`notify`](/configuration/tasks/#notify). Failures then go only to the bell in the Web UI and TUI, grouped
into one row with a count and a sparkline:

```diff
 [tasks.healthcheck-api]
 cron           = "*/5 * * * *"
 on_overlap     = "skip"
 timeout        = "30s"
-retry_attempts = 2
-retry_delay    = "30s"
-retry_backoff  = "linear"
 keep_runs      = 500
-notify         = ["slack-ops"]
```

This relies on [`global_notifiers`](/configuration/notify/#global_notifiers)
keeping its default `["inapp"]`.

## When RunWisp itself is down

A stopped daemon runs no checks and sends no alerts. To hear about that too,
set [`heartbeat_url`](/configuration/daemon/#heartbeat_url) to a dead man's
switch such as healthchecks.io. It alerts you when the daemon's pings stop.

## Probing internal services

Probe something that uses the service, not only its port. A database that
accepts connections but refuses queries passes a port check. For Postgres:

```bash
PGPASSWORD="$HEALTHCHECK_PG_PWD" psql --host=db.internal --username=healthcheck \
  --dbname=app_production --command='SELECT 1' --quiet --no-psqlrc > /dev/null
```

`pg_isready` is cheaper if you only need to know the server accepts connections.

When all you can check is an open port (a device you can't log in to), `nc` is
fine, but don't treat it as more than that:

```bash
nc -zw5 host.internal 5432 || { echo "host.internal:5432 not reachable"; exit 1; }
```
