Performance Troubleshooting: A Solid Checklist

1. Why systematic diagnosis beats aimless trial and error

A slow Elasticsearch cluster rarely has a single, obvious cause. Usually it is a combination of several factors: an unusually expensive query, a node with too little heap, an unfavorable shard distribution, or a temporary load spike overloading a thread pool. Anyone who randomly changes settings during a performance problem, for example increasing heap size or reducing replicas, without having understood the actual cause, risks briefly relieving the symptom while the real problem reappears at the next load increase.

A systematic performance diagnosis instead follows a fixed order: first a coarse overview of cluster health and resource usage, then a targeted search for slow operations via slow log and the hot threads API, followed by checking for structural bottlenecks such as a backed-up pending tasks queue or rejected requests in overloaded queues. This order ensures obvious, coarse problems are ruled out first, before time is invested analyzing more subtle causes.

This article combines the most important diagnostic tools into a practical checklist that can be worked through directly during a real incident, supplemented with concrete API calls and typical interpretations of the results.

2. Step one: checking cluster health and resource usage

The first thing to check for any performance problem is the cluster health status and the basic resource usage across all nodes. A status of yellow or red points to missing or unassigned shards, which often directly correlates with a performance drop, because affected requests have to fall back to fewer copies of the data or fail outright. The node statistics additionally show CPU usage, heap usage, disk I/O, and free disk space per node; often this coarse overview alone is enough to identify a single overloaded node as the cause.

A particularly important thing to check is disk watermarks: once a node reaches the high watermark for used disk space, Elasticsearch automatically blocks further shard allocation on that node, which can lead to an unfavorable, unbalanced load distribution across the entire cluster. This state is easy to miss because the cluster health status often still stays green, even though the actual resource distribution is already heavily unbalanced and individual nodes are significantly more loaded than others.

# Overall cluster health
curl -s "https://localhost:9200/_cluster/health?pretty" -u elastic:changeme

# Per-node resource usage: CPU, heap, disk
curl -s "https://localhost:9200/_cat/nodes?v&h=name,cpu,heap.percent,ram.percent,disk.used_percent" \
  -u elastic:changeme

# Check disk watermark thresholds currently configured
curl -s "https://localhost:9200/_cluster/settings?include_defaults=true&filter_path=**.disk.watermark*" \
  -u elastic:changeme

3. Slow log: identifying slow queries and indexing operations

The slow log records queries and indexing operations that exceed a configurable time threshold, separated into search and indexing slow logs, and further split by the query and fetch phases for search operations. Without an enabled slow log, it stays unclear which specific requests are responsible for a high performance load, since general cluster metrics only show the overall load, not which individual query is unusually expensive.

A sensible threshold for the warn level is often one to two seconds for the query phase, depending on the latency requirements of the specific application. Thresholds set too low flood the log with entries and make analysis harder, thresholds set too high leave relevant but not yet extremely slow queries undetected. A proven approach is to start with a conservative threshold and lower it step by step until the slow log delivers a manageable but meaningful volume of entries.

// Enable slow log thresholds for an index
PUT /products/_settings
{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.query.info": "1s",
  "index.search.slowlog.threshold.fetch.warn": "500ms",
  "index.indexing.slowlog.threshold.index.warn": "2s",
  "index.indexing.slowlog.level": "info"
}

# Tail the slow log directly on a data node to see live entries
tail -f /var/log/elasticsearch/production_index_search_slowlog.log

# Grep for the slowest recent queries above a threshold
grep "took\[" production_index_search_slowlog.log | sort -t'[' -k2 -rn | head -20

4. Hot threads API: understanding CPU load at thread level

While the slow log shows which queries are slow, the hot threads API shows which internal threads of a node are currently consuming the most CPU time, including a stack trace excerpt that reveals the operation currently being executed. This is especially valuable when CPU load on a node is high but it stays unclear whether the cause is search requests, indexing, merges, or internal management tasks such as cluster state updates.

The output of the hot threads API shows threads sorted by CPU share, with labels such as search, write, or [Lucene Merge Thread] that point directly to the cause. A high share of merge threads points to a write load that generates more segment merges than the node can comfortably process, while a high share of search threads points to expensive or too frequent search requests. The API can be run against individual nodes or all nodes and typically delivers a meaningful picture of the current CPU load within a few seconds.

# Hot threads across all nodes, sorted by CPU usage
curl -s "https://localhost:9200/_nodes/hot_threads?threads=5" -u elastic:changeme

# Hot threads for a single, specific node only
curl -s "https://localhost:9200/_nodes/node-data-03/hot_threads?threads=5" \
  -u elastic:changeme

5. Pending tasks: detecting a backlog on the master node

The master node processes cluster state updates such as creating indices, mapping changes, or shard assignments sequentially in an internal queue. Under normal conditions, this queue is empty or contains only a few entries processed within milliseconds. When pending tasks accumulate, it points to an overloaded master node, often caused by very many simultaneous index creations, an overly large cluster state object due to a very high number of indices and shards, or a master node that is additionally overloaded because it also serves as a data node.

A backed-up master node indirectly affects overall cluster performance, because every operation requiring a cluster state change, such as creating a new daily log index, gets stuck in the queue until the master node processes it. On very large clusters with thousands of indices, it is therefore a proven practice to run master nodes as dedicated, pure master nodes without a data role, so they are not simultaneously burdened by search requests or indexing.

# List pending cluster state tasks, if any are queued up
curl -s "https://localhost:9200/_cluster/pending_tasks?pretty" -u elastic:changeme

# Cluster state size can indirectly explain a slow master node
curl -s "https://localhost:9200/_cluster/state/_all?pretty" -u elastic:changeme | wc -c

6. Queue rejections: tracking down overloaded thread pools

Elasticsearch processes different operation types, such as search, write, and bulk indexing, in separate thread pools, each with a limited size and a limited queue. When more requests arrive than the thread pool and its queue can absorb, Elasticsearch rejects further requests with a TOO_MANY_REQUESTS error instead of buffering them indefinitely. These rejections, visible in the thread pool statistics as the rejected counter, are a direct signal of a structural capacity limit, not just a temporary slowdown.

A rising rejection counter in the bulk thread pool typically indicates that indexing clients are writing against the cluster with too large a bulk size or too much parallelism. The solution rarely lies in simply increasing the thread pool size, since that only shifts the underlying resource shortage elsewhere, but usually in reducing the bulk size, limiting the number of parallel indexing clients, or providing additional capacity through more nodes if the underlying load has permanently increased.

# Thread pool statistics with rejection counters per pool
curl -s "https://localhost:9200/_cat/thread_pool/write,search,bulk?v&h=node_name,name,active,queue,rejected" \
  -u elastic:changeme

# Watch rejections increase over time (run repeatedly during load)
watch -n 5 'curl -s "https://localhost:9200/_cat/thread_pool/write?v&h=node_name,rejected" -u elastic:changeme'

7. Shard size and count as a structural root cause

Beyond acute symptoms, an unfavorable shard size and shard count is one of the most common structural causes of chronic performance problems. Too many small shards create unnecessary management overhead, since every shard occupies its own Lucene file handles, segment metadata, and its own share of the cluster state. Overly large shards, on the other hand, slow down recovery after a node failure and make individual search requests slower, because a single shard can no longer be parallelized within one thread.

A common rule of thumb recommends shard sizes between 20 and 50 gigabytes for most use cases, though the actually optimal size depends on the specific query and indexing pattern. If a structural planning mistake in shard size is identified as the cause, the solution is usually a reindex into a newly created index with an adjusted number_of_shards setting, combined with index lifecycle management for time-based indices that automatically rotates new indices with an appropriate size.

8. Garbage collection and heap pressure as a special case

A frequently overlooked special case is performance degradation caused by frequent or long garbage collection pauses in the JVM. If a node's heap is chronically utilized above 75 to 85 percent, the garbage collector kicks in more and more often and with longer pauses, during which the affected node cannot process requests. These GC pauses show up in cluster performance as sporadic, hard-to-reproduce latency spikes that do not directly correlate with a specific query, which makes diagnosis harder without a targeted look at the GC logs.

A node's GC logs directly show the duration and frequency of garbage collection cycles, and long young-generation or even full GC pauses of several seconds are a clear signal of chronic heap pressure. The solution rarely lies in a blanket heap increase beyond the recommended 50 percent of available RAM or beyond 30 to 32 gigabytes, since that disables the JVM's compressed oops optimization and can paradoxically worsen performance, but usually in reducing the actual heap load through smaller aggregation windows, fewer simultaneous expensive queries, or additional nodes to distribute load.

9. The complete checklist at a glance

Combined, a fixed sequence emerges for every performance diagnosis: check cluster health and coarse resource usage, search the slow log for concrete slow operations, query the hot threads API under high CPU load, check pending tasks for a master node backlog, examine thread pool statistics for rejections, and, for chronic problems, consider shard size and GC behavior as structural root causes. This order covers both acute incidents and gradual, structural degradation.

The decisive advantage of this checklist over ad hoc diagnosis is reproducibility: every team member who follows the same sequence arrives at the same intermediate results and can document them in a way the team can follow, instead of improvising anew with every incident. A documented runbook containing this checklist as a concrete sequence of commands significantly reduces the time to root cause in a real incident, especially when it occurs under time pressure and possibly outside regular working hours.

10. Summary

Systematic performance troubleshooting for Elasticsearch and OpenSearch follows a fixed chain: cluster health and resources first, then the slow log for concrete slow operations, the hot threads API for CPU load at thread level, pending tasks for a master node backlog, thread pool rejections for capacity limits, and for chronic problems shard size and garbage collection behavior as structural root causes. This order reliably covers both acute and gradual problems.

The biggest lever is making this checklist available as a documented runbook before the next incident, instead of assembling it for the first time during a real one. A team that knows these tools and practices them regularly finds the cause of a performance problem in minutes instead of hours, which makes the decisive difference in production environments with direct business impact.

11. FAQ: Performance Troubleshooting Checklist