Shard Sizing: Finding the Right Number for Your Index

1. Why shard sizing is not a configuration detail

A shard is the smallest unit Elasticsearch can physically distribute and process in parallel, technically its own Lucene index. The number of primary shards is set when an index is created and cannot be changed directly afterward without rebuilding the index. This is exactly why shard sizing is one of the few Elasticsearch decisions that cannot simply be fixed with a configuration change once it turns out to be wrong.

Many teams stick with the Elasticsearch default of one primary shard per index, or set a high number like ten or twenty shards out of caution, without knowing the actual data volume. Both extremes cause problems: too few shards prevent parallelization for large data volumes, too many shards create management overhead that shows up directly in cluster state and heap usage on master nodes. Good shard sizing balances both against actual data volume and index growth.

The rule of thumb for shard sizing is not "more shards equals more performance", it orients around a target size per shard and the number of available data nodes. A shard that never grows past a few hundred megabytes wastes overhead. A shard that reaches two hundred gigabytes becomes a bottleneck for recovery and rebalancing.

2. The oversharding trap: many small shards

Oversharding usually creeps in gradually: a daily index for log data gets created with five primary shards because that is what an old piece of documentation said, but only two gigabytes of data actually accumulate per day. After a year, three hundred sixty five indices exist with five shards each, over eighteen hundred shards for a data volume that a hundred well-sized shards could easily handle. Each of these shards costs cluster state metadata, open file handles, and its own Lucene segment merge cycle.

The consequences of oversharding show up first on the master nodes: cluster state grows with every shard allocation, its publication to all nodes becomes slower, and shard allocation takes noticeably longer after a node restart. On the search side, latency degrades because every search request has to be sent to all relevant shards and their results merged, even if a single shard contains only a few documents. With overly aggressive upward shard sizing, you pay this coordination overhead for practically no parallelization gain.

3. The target size per shard: between 10 and 50 GB

A practical guideline for shard sizing has established itself as a target range of ten to fifty gigabytes per shard, for most general use cases with text and log data. Within this range a shard stays small enough to reallocate or recover in a reasonable time, and large enough to keep the per-shard overhead small relative to the actual data volume. For time-based log indices with ILM, the same rule is the starting point for the rollover criterion.

The exact number within this range depends on the use case: full-text search with many aggregations tends to benefit from smaller shards around twenty gigabytes, because aggregation work parallelizes well, while pure log storage with rare, simple queries also works fine with fifty gigabytes per shard. It matters to not set this value once and forget it, but to regularly check it against actual data growth, especially for indices that grow continuously instead of rotating on a time basis.

4. Correctly placing primary and replica shards

Primary shards are the set that defines shard sizing, they contain the original data and are fixed when the index is created. Replica shards are complete copies of the primary shards on other nodes, they serve fault tolerance and additionally increase read throughput because search requests can also be answered against replicas. The number of replicas can be changed at any time without reindexing, unlike the primary shard count.

A common misunderstanding: replicas count toward capacity planning just like primaries, an index with five primary shards and one replica occupies ten shard slots in the cluster. When doing shard sizing, the total count including replicas should always be considered, not just the primary count, since it consumes the same storage, heap, and file handles on the data nodes.

PUT /orders-2026.07
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "order_id": { "type": "keyword" },
      "customer_email": { "type": "keyword" },
      "created_at": { "type": "date" },
      "total_amount": { "type": "scaled_float", "scaling_factor": 100 }
    }
  }
}

// 3 primary shards + 1 replica each = 6 shard slots total across the cluster

5. Calculating shard count in advance

For new indices, the appropriate primary shard count can be estimated in advance if the expected data volume is known: expected total index size divided by the target size per shard, rounded up to a practical number. An index expected to reach two hundred gigabytes, with a target size of forty gigabytes per shard, requires five primary shards mathematically. This calculation is an approximation, not an exact value, because data growth rarely follows a linear pattern.

For indices with uncertain growth projections, it is usually safer for shard sizing to start with a somewhat smaller shard count and grow it via the split API if needed, rather than planning oversized from the start. The reason: empty or barely filled shards cause the same management overhead as full shards, so an overly generously estimated index ties up unnecessary cluster resources from day one.

GET _cat/shards/orders-2026.07?v&h=index,shard,prirep,state,docs,store

// Store column shows actual size on disk per shard, compare against target
index            shard prirep state   docs      store
orders-2026.07   0     p      STARTED  4823012   38.2gb
orders-2026.07   0     r      STARTED  4823012   38.2gb
orders-2026.07   1     p      STARTED  4801556   37.9gb
orders-2026.07   1     r      STARTED  4801556   37.9gb
orders-2026.07   2     p      STARTED  4790112   37.6gb
orders-2026.07   2     r      STARTED  4790112   37.6gb

6. Undersharding: when shards grow too large

The opposite of oversharding is equally problematic: a single primary shard that grows to two hundred gigabytes or more becomes a bottleneck. Recovery after a node failure takes time proportional to shard size, because the entire shard has to be copied from another node before it becomes available again. During this recovery phase, less redundancy is available, which increases the risk of data loss on a second failure.

Write load also suffers from shards that are too large: segment merges on a very large shard tie up more I/O and CPU at once, causing noticeable latency spikes during indexing. Correct shard sizing therefore means not only avoiding oversharding, but equally preventing an index from growing organically past the recommended upper limit without anyone intervening.

7. Resharding without downtime via the reindex API

Since the primary shard count is fixed after creation, the classic way to correct wrong shard sizing goes through the reindex API: a new index with a corrected shard count is created, all documents are copied from the old to the new index via _reindex, and an alias is then repointed. As long as write access happens through an alias instead of directly against the index name, this switch happens without downtime for reading and writing clients.

With ongoing write traffic during the reindex, a two-stage approach is needed: first the initial reindex of the historical data, then a second, incremental reindex run with a time filter on newly written documents, before the alias is finally switched over. This pattern is standard for any larger shard sizing rebuild of production indices.

// Step 1: create the new index with corrected shard count
PUT /orders-2026.07-v2
{
  "settings": { "number_of_shards": 5, "number_of_replicas": 1 }
}

// Step 2: copy all existing documents into the new index
POST /_reindex?wait_for_completion=false
{
  "source": { "index": "orders-2026.07" },
  "dest": { "index": "orders-2026.07-v2" }
}

// Step 3: atomically swap the alias, zero downtime for clients
POST /_aliases
{
  "actions": [
    { "remove": { "index": "orders-2026.07", "alias": "orders-current" } },
    { "add": { "index": "orders-2026.07-v2", "alias": "orders-current" } }
  ]
}

8. Split and shrink APIs as an alternative

For the special case of undersharding, Elasticsearch offers the split API: it divides an existing index into more shards without documents having to be manually copied over, provided the target shard count is a multiple of the original count. For oversharding, the shrink API exists in the other direction, it reduces the shard count of a write-blocked index. Both operations are faster than a full reindex operation because they work at the Lucene segment level instead of rewriting document by document.

The restriction of both APIs: the source index must first be set to read-only, which for actively written production indices is usually only practical for time-based, already closed indices. For currently written indices, the reindex API with alias switching therefore remains the usual path, while split and shrink integrate especially well into ILM policies for closed, older indices.

// Mark the source index read-only before splitting or shrinking
PUT /orders-2026.06/_settings
{
  "settings": { "index.blocks.write": true }
}

// Split into more shards, target must be a multiple of the source count
POST /orders-2026.06/_split/orders-2026.06-split
{
  "settings": { "index.number_of_shards": 6 }
}

// Shrink into fewer shards for an oversharded, closed index
POST /orders-2026.06/_shrink/orders-2026.06-shrunk
{
  "settings": { "index.number_of_shards": 1 }
}

9. Monitoring shard size continuously

Static shard sizing at index creation time is not enough once growth patterns change. The _cat/shards API with the store field shows the current on-disk size for every shard and makes visible which shards are approaching the upper limit or staying well below the target value. A regular, automated check of these values, for example weekly via script, catches unwanted developments before they become an acute problem.

For time-based indices with ILM, the rollover condition max_primary_shard_size is the direct tool for enforcing shard sizing automatically: a new index is created as soon as a primary shard reaches the configured size, regardless of how much time has passed. That decouples shard size from unpredictable fluctuations in daily data volume and reliably keeps every single shard within the target range.

PUT _ilm/policy/orders-rollover-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "40gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

// Rollover fires on whichever condition is met first, size or age

10. Summary

Good shard sizing moves within a target range of ten to fifty gigabytes per shard and is roughly calculated in advance against expected data volume, not set by gut feeling or old documentation values. Oversharding burdens cluster state and master nodes, undersharding slows down recovery and creates write latency spikes. Since the primary shard count is fixed after creation, the path to correction goes through reindex with alias switching, or through split and shrink APIs for read-only indices.

The most sustainable approach is to treat shard sizing not as a one-time decision but as a continuously monitored parameter: with max_primary_shard_size in ILM policies for time-based indices and regular size checks for everything else. That keeps every index within the target range without manual intervention as data volume grows.

11. FAQ: Shard Sizing in Elasticsearch