Metric alerts

What you get#

• Threshold rules — condition on cpu, memory, requests, or egress for a deployment: metric, comparison, threshold, and how long it must hold.
• Rolling-window evaluation — a rule fires only once the condition has held continuously for its window, not on a single noisy sample.
• Three states — ok, firing, or nodata, so a missing deployment or a gap in metrics is never confused with a real breach.
• Renotify — get pinged again on a schedule while a rule is still firing, not just once.
• Delivery via notification channels — no separate delivery config; wire a channel to alert.trigger / alert.resolve like any other event.
• 30-day event history — every state transition is recorded for the detail page.

Create a rule#

From the console, open Alerts and click Create rule. Or use the CLI:

deploys alert create \
  --project acme \
  --name web-cpu-high \
  --location gke.cluster-rcf2 \
  --deployment web \
  --metric cpu \
  --op ">=" \
  --threshold 90 \
  --for 10 \
  --renotify 60

This watches the web deployment’s CPU usage and fires once it has averaged ≥ 90% of its limit for 10 straight minutes, re-notifying every 60 minutes while it stays firing.

Fields#

Metric vocabulary#

cpu and memory are computed the same way the Metric tab’s chart lines are — avg(usage) / avg(limit) per one-minute bucket — so the threshold you set lines up visually with what you see on the metrics chart.

When a rule fires#

A rule evaluates every minute over a rolling window of the last for minutes, not a single instant:

• firing — every one-minute bucket present in the window satisfies the condition (metric <op> threshold), and at least 80% of the expected buckets are present. A single missed collector minute doesn’t reset the clock.
• nodata — fewer than 20% of the expected buckets are present — the deployment is stopped, deleted, or (for cpu/memory) has no limit set.
• ok — otherwise.

nodata never fires and never resolves an active alert: a rule that’s already firing stays firing through a data gap, and only clears once the metric is genuinely back under (or over, for <=) the threshold. This keeps “no data” and “deployment is down” — which is deployment.health’s job — from double-paging the same incident, and keeps a flaky collector minute from silently clearing a real one.

Because evaluation is windowed, resolving a firing alert takes one extra tick after the first good minute enters the window — a small amount of built-in hysteresis so a single clean sample mid-incident doesn’t flap the alert closed and back open.

Status#

The console list and detail pages show a rule’s current status, last evaluated value, and — while firing — how long it’s been firing. From the CLI:

deploys alert list --project acme
deploys alert get  --project acme --name web-cpu-high

Renotify#

By default (renotify: 0) a rule notifies only on transitions — the moment it starts firing and the moment it resolves — and stays quiet in between, however long the incident runs. Set --renotify to a number of minutes (10–1440) to also re-send alert.trigger on that cadence while the rule is still firing, for teams that want a periodic reminder rather than a single page.

# re-notify every 30 minutes while firing
deploys alert update --project acme --name web-cpu-high --renotify 30

# transitions only
deploys alert update --project acme --name web-cpu-high --renotify 0

Delivery: alert.trigger and alert.resolve#

An alert rule carries no delivery config of its own — it reuses notification channels entirely. Subscribe a channel to alert.trigger and alert.resolve (or alert.* to also see rule config changes, since create/update/delete on a rule are ordinary audited changes like any other resource):

deploys notification create --project acme --name alerts-discord \
  --type discord \
  --url https://discord.com/api/webhooks/123/abc \
  --event alert.trigger --event alert.resolve

• alert.trigger carries outcome failure (red in Discord).

• alert.resolve carries outcome success (green in Discord).

• The message is a one-line summary of the condition and the value that crossed it, e.g.:

  web: cpu >= 90% for 10m (current 94.2%)
  

History#

Every state transition (trigger, resolve, and each renotify) is recorded with its value, kept for 30 days, and shown on the rule’s detail page alongside a link to the deployment’s metrics chart:

deploys alert events --project acme --name web-cpu-high --limit 50

Limits#

Using the API directly#

Every console action and CLI command is a thin wrapper over the API: alert.create, alert.update, alert.get, alert.list, alert.delete, and alert.events (recent transitions). These are also exposed to AI assistants through the MCP server.

Permissions#

Grant these on a role like any other permission. An alert rule’s config carries nothing sensitive, so unlike notification channels, alert.get / alert.list are grantable to public principals (allUsers / allAuthenticatedUsers) like most read permissions.