Mapping Design Fundamentals: Planning Your Elasticsearch Index Correctly

1. What mapping is and why it is the most important decision

The mapping of an Elasticsearch index is the schema that determines how every field of a document is stored, indexed and made searchable. Unlike a classic SQL table, an Elasticsearch index is not a rigid grid of columns, it is a JSON document store that still needs clear type rules. Mapping defines exactly these rules: which field is full text, which one is an exact comparison value, which one is a number, which one is a date. Without deliberate planning, Elasticsearch decides these questions itself, and the automatic decision is not always the right one for the actual use case.

Practice keeps showing the same sequence of events: a team indexes the first test data, everything works, the index goes to production, and weeks later it turns out a price aggregation does not work or sorting by product name fails. The cause is almost always the mapping that was auto generated on the first document and never consciously reviewed. This article shows which field types exist, how to assign them correctly, and why investing in deliberate mapping pays off after the very first week in production.

2. Field types at a glance: text, keyword, numeric, date

Elasticsearch has a whole range of field types, but four categories cover most practical cases. The type text is meant for full text search: the value passes through an analyzer that splits it into individual tokens, so a search for a partial term inside a longer sentence works. The type keyword instead stores the value unchanged as a single token and is suited for exact comparisons, filters, sorting and aggregation, for example status fields, IDs or tags. This distinction is the most important building block of any mapping design.

Numeric types such as long, integer, short, float and double enable range queries, mathematical aggregations and efficient sorting. The type date stores timestamps internally as milliseconds since the Unix epoch and accepts various format strings on indexing, which can be explicitly configured in the mapping. In addition there is boolean for truth values, object and nested for nested structures, as well as specialized types like geo_point for coordinates. Each of these decisions in the mapping has a direct effect on which queries are possible later.

A common misconception: many developers think the field type is just a storage optimization. In reality the type in the mapping also determines which query clauses work at all. A match query on a keyword field rarely returns the desired result because no analyzer has prepared the tokens. Conversely, a term query on a text field often returns unexpected hits because it compares against the individual tokens produced by the analyzer, not against the original value.

PUT /products
{
  "mappings": {
    "properties": {
      "sku":          { "type": "keyword" },
      "name":         { "type": "text" },
      "description":  { "type": "text" },
      "price":        { "type": "double" },
      "stock_count":  { "type": "integer" },
      "created_at":   { "type": "date", "format": "strict_date_optional_time||epoch_millis" },
      "in_stock":     { "type": "boolean" },
      "brand":        { "type": "keyword" }
    }
  }
}

3. Why text and keyword are fundamentally different

The difference between text and keyword is the point where mapping design is most often misunderstood. A text field runs through an analyzer during indexing that lowercases the value, strips punctuation and splits the text into individual words. The value "Wireless Headphones Pro" becomes the tokens "wireless", "headphones" and "pro", each individually searchable. That is exactly what makes full text search possible, but it makes the field unsuitable for exact comparisons or sorting, because the original string no longer exists in a sortable form.

A keyword field, in contrast, stores "Wireless Headphones Pro" as exactly that one string, without splitting it. A filter query for exactly this value works reliably, sorting on the field returns alphabetically correct results, and an aggregation counts each unique string separately. Anyone trying to sort on a text field will either get an error or need to fall back on an additional keyword sub field that exists exactly for this purpose.

4. Choosing numeric and date types correctly

For numeric fields, a second look at the size of the chosen type pays off. A long field consumes more storage than an integer field and is usually overkill for a value like a stock count. On large indices with millions of documents this difference adds up noticeably in memory usage and query speed. Newer Elasticsearch versions also offer the type scaled_float, which stores floating point values as a scaled integer and is more efficient than double for prices with a fixed number of decimal places.

For the date type the format attribute in the mapping is decisive. Without an explicit format, Elasticsearch accepts several default formats at once, which can lead to inconsistent values when different data sources deliver different formats. An explicitly set format such as strict_date_optional_time||epoch_millis in the mapping forces every incoming value into a consistent pattern and prevents a malformed timestamp from either failing the entire indexing request or being silently misinterpreted.

# Inspect the current mapping of an existing index
curl -s -X GET "https://localhost:9200/products/_mapping?pretty" \
  -u elastic:changeme

# Check a single field's mapping only
curl -s -X GET "https://localhost:9200/products/_mapping/field/price?pretty" \
  -u elastic:changeme

5. Why mapping changes are so difficult once live

The core reason mapping design should happen before indexing starts lies in Elasticsearch's internal architecture: an existing field cannot later be changed from one type to another. Trying to change a text field to keyword afterward fails with a clear error, because the underlying Lucene segment structure was already written with the original type. This immutability is not a limitation of Elasticsearch, it is a direct consequence of how inverted indices are physically built.

What is possible afterward in a mapping is additive change: new fields can be added at any time, as long as they did not exist before. Certain parameters like ignore_above on keyword fields can also be updated in some cases. But any change that touches the underlying data type strictly requires a new index with a corrected mapping and a subsequent reindex operation. This makes clear why the first mapping decision carries so much weight: a mistake here is not a bug fix, it is a data migration.

6. Explicit mapping instead of dynamic detection

Without an explicit definition, Elasticsearch automatically creates a mapping on the first document using its own heuristics. A string value is typically indexed both as text and as a keyword sub field, a number as long or float, a recognizable date format as date. This automatic detection works acceptably for simple cases, but it regularly makes decisions that are suboptimal for the actual use case, for example when a product ID gets detected as a number instead of a keyword and loses its leading zeros as a result.

The robust approach is to define the mapping explicitly with a PUT request before the first document. This costs a bit more time during index setup, but it prevents exactly the surprises that later cause reindexing effort. A good workflow is to start with a handful of realistic test documents, inspect the auto generated mapping via GET _mapping, use it as a starting point, and then consciously correct it before the production data import begins.

7. Index templates: fixing mapping for future indices

For use cases with time based indices, such as daily log indices, it would be impractical to set the mapping manually every day. This is where index templates come in: a template defines a mapping that is automatically applied to every newly created index whose name matches a defined pattern. Every index matching the pattern logs-app-* then automatically gets the same mapping without an operator having to intervene.

Composable index templates, available since Elasticsearch 7.8, additionally allow assembling mapping building blocks from multiple component templates. A shared base with standard fields like @timestamp can be combined with application specific extensions without redundancy in the mapping. This significantly reduces copy paste errors and makes changes to shared fields maintainable in a single place.

PUT /_index_template/logs-app-template
{
  "index_patterns": ["logs-app-*"],
  "priority": 200,
  "template": {
    "settings": { "number_of_shards": 1, "number_of_replicas": 1 },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level":      { "type": "keyword" },
        "message":    { "type": "text" },
        "service":    { "type": "keyword" },
        "trace_id":   { "type": "keyword" }
      }
    }
  }
}

8. Reindexing as the way out of a planning mistake

When an existing mapping needs to be corrected after all, there is no way around a reindex. The process: a new index with a corrected mapping is created, the reindex API copies all documents from the old index into the new one, and an alias is switched so the application points to the new index without downtime. This alias swap is the crucial trick that makes reindexing in production plannable, instead of forcing hard downtime.

It is important not to underestimate the reindex process on large indices: with several hundred million documents, a full reindex can take hours and bind significant cluster resources. The reindex API therefore supports slices for parallelization and a wait_for_completion=false parameter to run the operation asynchronously in the background and query progress separately. Anyone practicing clean mapping design from the start reduces how often such a costly operation becomes necessary at all.

POST /_reindex?wait_for_completion=false
{
  "source": { "index": "products_v1" },
  "dest":   { "index": "products_v2" },
  "conflicts": "proceed"
}

GET /_tasks/<task_id>

POST /_aliases
{
  "actions": [
    { "remove": { "index": "products_v1", "alias": "products" } },
    { "add":    { "index": "products_v2", "alias": "products" } }
  ]
}

9. Mapping design in practice: a complete example

A realistic example makes the principles tangible. A product catalog for an online shop needs at least the following field categories in the mapping: a unique, exactly comparable SKU as keyword, a searchable product name as text with an additional keyword sub field for sorting, a price as double or scaled_float, a stock count as integer, and a creation date as date with an explicit format. This interplay shows that good mapping design is not a single trick, it is the consistent application of several small decisions.

Before a production rollout, a test run pays off: load a few hundred real sample documents into a test index, try out every planned query against this mapping, especially sorting, aggregations and range filters, and only then create the final index for the production import. This test run costs about an hour and can save days of later reindexing if a field type turns out to be wrong.

For Magento operators using the built in Elasticsearch catalog index, the same principle applies at a different layer: custom attributes configured with the searchable and filterable flags on the product attribute directly affect the generated mapping of the catalog index. An attribute mistakenly marked as searchable text instead of a filter value produces the same problems as a poorly planned mapping in a custom index, just hidden behind the Magento abstraction layer.

This table is not a replacement for complete mapping documentation, but it shows the basic rule: as soon as sorting or aggregation is needed, an exact field type such as keyword, a numeric type, or date is required. Full text search always belongs to text, never to one of the exact types. Anyone following this basic rule in mapping design avoids most of the query errors observed later.

10. Summary

Good mapping design does not begin at the first query, it begins at the first document. The field types text, keyword, numeric types and date cover most use cases, but they must be chosen deliberately, because Elasticsearch's dynamic detection does not always make the right call. The immutability of existing field types turns every later correction into a reindex operation, which is why planning before the production import represents the biggest leverage point.

Index templates automate consistent mapping across many indices, while the reindex API combined with an alias swap offers a plannable escape route when a correction becomes necessary anyway. Anyone who knows these building blocks and deliberately designs a mapping before the first data import avoids the typical cycle of test index, surprise, and emergency reindex that unnecessarily costs time on many projects.

11. FAQ: Mapping Design Fundamentals