Setting Up Cluster Health Monitoring and Alerting Properly

1. Why green/yellow/red alone is not enough

The _cluster/health endpoint returns the status green, yellow or red, the best-known figure for cluster health monitoring. Many teams base their entire alerting exclusively on this single value, which creates a deceptive sense of safety: the status often only switches to yellow or red once a problem already affects availability, not when it originates. A cluster running for hours with steadily rising heap pressure formally stays green until the first node crashes.

Effective cluster health monitoring therefore needs additional, more fine-grained metrics that point to brewing problems before they affect the overall status. Unassigned shards, heap usage, queue rejections and node availability are four metrics that together provide a much earlier picture than the pure health status. The goal is to react to trends, not only to outages that have already happened.

This article ranks the most important signals for cluster health monitoring by urgency and shows concrete API calls to query them automatically and translate them into alerting rules, regardless of whether Prometheus, the Elastic Stack's own Watcher, or an external monitoring tool is used.

2. The _cluster/health status in detail

The status green means that all primary and replica shards are assigned. Yellow means all primary shards are available, but at least one replica shard could not be assigned, usually because not enough data nodes exist for the configured replica count. Red means at least one primary shard is unavailable, which for the affected indices means direct data loss for read requests as long as the condition persists.

A common mistake in cluster health monitoring is treating yellow as generally uncritical. A single test index without replica configuration that is permanently yellow differs fundamentally from a production index that suddenly turns yellow because a data node failed. Alerting should therefore not just react to the global status, but to status changes and to per-index health, available via _cluster/health?level=indices.

GET _cluster/health?level=indices

// Global status can be yellow while individual indices are still red,
// per-index detail is essential for correct alert prioritization
{
  "cluster_name": "production-logs",
  "status": "yellow",
  "number_of_nodes": 9,
  "active_primary_shards": 142,
  "unassigned_shards": 3,
  "indices": {
    "orders-2026.07": { "status": "green", "number_of_shards": 3, "unassigned_shards": 0 },
    "sessions-2026.07": { "status": "yellow", "number_of_shards": 5, "unassigned_shards": 3 }
  }
}

3. Unassigned shards: causes and diagnosis

Unassigned shards are one of the most important individual metrics in cluster health monitoring, because they almost always point to a concrete, diagnosable problem. The most common causes: a node has failed and its shards need reallocation, the disk watermark threshold has been exceeded on a node so Elasticsearch stops distributing new shards there for safety, or allocation filter rules prevent a valid assignment, for example through misconfigured shard allocation awareness.

The _cluster/allocation/explain API is the central diagnostic tool for unassigned shards: it returns the exact reason a specific shard could not be assigned, instead of forcing a guess. A good cluster health monitoring setup calls this API automatically as soon as the unassigned shard count rises above a zero threshold, and attaches the response directly to the alert to save diagnosis time in an incident.

GET _cluster/allocation/explain
{
  "index": "sessions-2026.07",
  "shard": 2,
  "primary": false
}

// Response reveals the concrete blocking reason instead of forcing a guess
{
  "index": "sessions-2026.07",
  "shard": 2,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "NODE_LEFT",
    "at": "2026-07-24T03:14:22.000Z"
  },
  "allocate_explanation": "cannot allocate because a previous copy of the primary shard existed but can no longer be found"
}

4. Heap pressure as an early warning signal

Heap usage is the most important early warning metric in cluster health monitoring, because it often rises long before a node actually becomes unstable. The _nodes/stats API returns heap usage per node as a percentage. A rough alerting threshold that has established itself is a warning at seventy five percent sustained usage and a critical value from eighty five percent, where "sustained" is the key word: short spikes after a garbage collection cycle are normal, a value that does not drop even after several GC cycles is not.

For resilient cluster health monitoring, a single instantaneous value is not enough. A moving average over several minutes, combined with the number of garbage collection cycles in the same period, is more meaningful. If GC frequency rises together with heap usage, that is a much stronger signal for a genuine memory problem than a single high percentage.

5. Queue rejections from thread pools

Every thread pool in Elasticsearch, for example for search, write or bulk, has a limited queue length. When more requests arrive than the pool can process and the queue can hold, additional requests get rejected with a rejection error instead of waiting indefinitely. For cluster health monitoring, these rejections are a direct, unambiguous signal that the cluster is receiving more load than it can currently process.

Unlike heap pressure, which builds up gradually, queue rejections are binary: a request is either rejected or not, and every rejection means a concrete error for the calling client. An alert on rejected > 0 in the relevant thread pool is therefore one of the most direct alerting rules possible, with no room for threshold interpretation.

GET _nodes/stats/thread_pool/write,search,bulk

// A non-zero "rejected" counter means clients received real errors,
// this is one of the clearest possible alert conditions
{
  "nodes": {
    "abc123": {
      "thread_pool": {
        "write": { "threads": 8, "queue": 42, "active": 8, "rejected": 17 },
        "search": { "threads": 13, "queue": 3, "active": 5, "rejected": 0 },
        "bulk": { "threads": 8, "queue": 0, "active": 2, "rejected": 0 }
      }
    }
  }
}

6. Further metrics for resilient monitoring

Beyond the three core metrics, further figures belong in complete cluster health monitoring: disk watermark usage per node, since Elasticsearch actively moves shards off a node once the high watermark is exceeded, causing unassigned shards if no target with enough space exists. The number of pending cluster state tasks, via _cluster/pending_tasks, shows whether master node operations such as shard allocations are backing up, an indicator of an overloaded master.

Segment merge counts and indexing latency are also relevant for data-heavy clusters, though more for performance than for pure availability monitoring. To get started with cluster health monitoring, it is enough to first focus on health status, unassigned shards, heap pressure and queue rejections, and then gradually extend monitoring with further metrics depending on which incidents actually occur in operation.

# elasticsearch.yml - default disk watermark thresholds, adjust with care
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
cluster.routing.allocation.disk.watermark.flood_stage: 95%

# Above flood_stage, indices on the affected node are forced read-only
# until free disk space drops back below the threshold again

7. Prioritizing alerting rules correctly

Not every deviation in cluster health monitoring deserves the same reaction speed. A red cluster status with affected primary shards needs an immediate, round-the-clock response, while a single short-lived heap spike on one node usually warrants a lower-priority note for the next working day. A three-tier prioritization, for instance critical, warning and informational, prevents alert fatigue and ensures truly urgent messages do not drown in a flood of less relevant notifications.

The time dimension also matters for sustainable cluster health monitoring: most metrics should not react to a single data point but to sustained conditions over a defined time window. A brief heap spike during a scheduled snapshot operation is normal, the same value sustained over fifteen minutes is not. This distinction reduces false alarms significantly without missing real problems.

8. Tools for cluster health monitoring

There are several established paths for implementing cluster health monitoring technically: the Elasticsearch exporter for Prometheus exposes the relevant metrics in a format that plugs directly into Grafana and links to Alertmanager. Alternatively, the Elastic Stack itself offers an integrated solution with Kibana Stack Monitoring and Watcher, requiring no additional infrastructure but tied to a license tier.

For smaller setups, a simple script that periodically queries the relevant APIs and triggers a notification once a threshold is exceeded, for example via webhook to Slack or PagerDuty, is often enough. More important than the chosen tool for cluster health monitoring is that metrics are captured consistently and alerting thresholds are regularly checked against actual cluster size and workload, instead of leaving values set once unchanged forever.

9. Escalation levels and runbooks

An alert alone does not solve a problem, which is why good cluster health monitoring always comes with a documented runbook: for every critical alert type it should be recorded which diagnostic steps to run first, for example _cluster/allocation/explain for unassigned shards, and which immediate measures are options, such as manually triggering shard reallocation or temporarily raising a disk watermark in an emergency situation.

Escalation levels should clearly define when an automated alert is enough and when a human needs to be notified, ideally with different channels depending on urgency: informational notes in a monitoring dashboard, warning signals in a team chat, critical alerts in an on-call system with guaranteed delivery. This structure is what makes cluster health monitoring genuinely actionable, instead of just collecting data no one sees in time.

PUT _watcher/watch/critical_unassigned_shards
{
  "trigger": { "schedule": { "interval": "1m" } },
  "input": {
    "http": { "request": { "host": "localhost", "port": 9200, "path": "/_cluster/health" } }
  },
  "condition": {
    "compare": { "ctx.payload.unassigned_shards": { "gt": 0 } }
  },
  "actions": {
    "notify_oncall": {
      "webhook": {
        "method": "POST",
        "host": "hooks.example.com",
        "path": "/oncall-critical",
        "body": "Unassigned shards detected, run _cluster/allocation/explain immediately"
      }
    }
  }
}

10. Summary

Effective cluster health monitoring goes beyond the simple green/yellow/red status and watches unassigned shards, heap usage and queue rejections as central early warning signals. The _cluster/allocation/explain API delivers concrete diagnostic reasons instead of guesswork, while thread pool statistics show directly when the cluster receives more load than it can process. Sustained conditions over a time window are more meaningful than single instantaneous values.

The final, often neglected building block is prioritization: not every alert needs an immediate response, but every critical alert type needs a documented runbook with clear diagnosis and response steps. Anyone who builds cluster health monitoring this way catches problems before they become outages, instead of only reconstructing afterward what went wrong.

11. FAQ: Cluster Health Monitoring and Alerting