Analyzers and Tokenizers in Detail

1. What an analyzer is and why every search needs one

An analyzer is the component in Elasticsearch that transforms a text string into a sequence of searchable tokens. Without an analyzer, every text field would only be comparable as an exact string, just like a keyword field, and searching for a single word inside a longer sentence would be impossible. The analyzer runs twice: once while indexing a document to produce the stored tokens, and once while executing a search query to turn the search terms into comparable tokens.

The fact that a search for "running" finds a document containing "Running" despite the case difference is not magic, it is the result of an analyzer normalizing both values to the same token. Likewise it is no coincidence when a search unexpectedly fails because index time and search time produced different tokens. This article breaks the analyzer down into its three building blocks and shows how to test its behavior deliberately instead of guessing.

2. The analyzer pipeline: character filter, tokenizer, token filter

An analyzer is not a single function, it is a three stage pipeline. First the raw text passes through zero or more character filters, which modify the string before splitting, for example stripping HTML tags or replacing certain characters. Next, exactly one tokenizer is applied, splitting the cleaned text into individual tokens, usually at word boundaries. Finally these tokens pass through zero or more token filters, which modify, remove or add individual tokens, for example through lowercasing, stopword removal or stemming.

This order is fixed and cannot be swapped: character filters always operate on the raw string, the tokenizer always splits exactly once, and token filters always operate on the already produced token list. This clear separation makes the pipeline predictable and allows each of the three building blocks to be swapped independently without affecting the other two. An analyzer is ultimately nothing more than a named combination of zero or more character filters, exactly one tokenizer, and zero or more token filters.

// Pipeline order is always fixed:
// raw text -> char_filter[] -> tokenizer -> token_filter[] -> final tokens

GET /_analyze
{
  "char_filter": ["html_strip"],
  "tokenizer": "standard",
  "filter": ["lowercase", "stop"],
  "text": "<p>The Quick Brown Fox jumps!</p>"
}

// Result tokens roughly: quick, brown, fox, jumps

3. Character filters: cleaning text before it is split

Character filters operate on the unmodified input string before any splitting happens. The built in filter html_strip removes HTML markup and decodes HTML entities, which is often needed for content coming from a CMS or rich text editor, so tags like <strong> do not accidentally end up as part of a token. The mapping filter replaces configurable character sequences with others, for example turning "&" into "and" before the tokenizer runs.

The pattern_replace filter allows regular expressions for more complex replacements, for example removing phone number formatting characters before indexing. It is important to use character filters sparingly: every additional filter costs processing time on every indexed document and every search request, and overly aggressive cleaning can remove relevant information from the text before the tokenizer even gets a chance to run.

4. Tokenizers: splitting text into tokens

The tokenizer is the heart of every analyzer and appears exactly once per pipeline. The standard tokenizer implements the Unicode text segmentation rules and splits text at word boundaries, removing punctuation while sensibly handling periods in abbreviations and email addresses. For most European languages it returns usable results without further configuration.

The whitespace tokenizer splits only on whitespace and keeps punctuation as part of the tokens, which can be useful for code examples or technical identifiers where punctuation carries meaning. The ngram and edge_ngram tokenizers split words into overlapping character sequences of fixed length and form the basis for autocomplete and substring search, because they allow matches inside a word, where a standard tokenizer only returns whole words as tokens. Finally the keyword tokenizer returns the entire input string as a single token and is often used in combination with token filters for controlled normalization when no real word splitting is desired.

GET /_analyze
{
  "tokenizer": "edge_ngram",
  "text": "search"
}
// Tokens: s, se, sea, sear, searc, search
// Useful base for autocomplete-style prefix matching

5. Token filters: refining and normalizing tokens

Token filters run after the tokenizer and can be chained as many times as needed, with each filter receiving the output of the previous one as input. The lowercase filter is by far the most common and ensures that "Search" and "search" map to the same token. The stop filter removes common filler words like "and", "the", "of" from the token stream to reduce index size and avoid relevance noise, but it should be used carefully, because stopwords can be semantically important in some phrase searches.

Stemming filters like stemmer with a language configuration reduce words to their root form, so that "running", "runs" and "ran" can map to a shared token and a search for one form also finds the others. The synonym filter expands tokens with configured synonyms, so a search for "laptop" also finds documents containing "notebook". These filters noticeably increase recall, but if configured too aggressively they can also introduce unwanted matches, which is why testing with real search terms is indispensable.

6. The standard analyzer and its limits

Without an explicit configuration, Elasticsearch uses the standard analyzer for every text field, consisting of the standard tokenizer and the lowercase token filter, with no character filters and no stemming. It works decently for generic, multilingual text, but is rarely optimal for specialized use cases. German compound words like "Bahnhofsvorplatz" (station forecourt), for example, are treated as a single token, so a search for "Bahnhof" (station) returns no hit, even though the word would make sense in context.

For such cases, Elasticsearch offers built in, language specific analyzers like german, which come with additional token filters for normalization, stemming, and in some cases compound word splitting. Switching from the standard analyzer to a language specific analyzer is often the quickest lever to noticeably improve search quality in a given language, without assembling a pipeline from scratch.

7. Assembling a custom analyzer

When neither the standard analyzer nor the built in language specific variants fit, a custom analyzer can be freely assembled from the three building blocks in the index mapping. This is the point where Elasticsearch shows its full flexibility: a team could build an analyzer that strips HTML tags, splits with the standard tokenizer, lowercases everything, removes German stopwords and then applies stemming, all in a single named definition.

The important thing with custom analyzers is to never design them in isolation, always in the context of real search terms. An overly aggressively configured analyzer with many stemming and synonym filters increases the number of hits, but often lowers precision, because too many thematically unrelated documents also match. A good starting point is always to begin with the standard analyzer and add individual filters deliberately, instead of designing a complete pipeline from zero.

PUT /articles
{
  "settings": {
    "analysis": {
      "analyzer": {
        "german_content": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "german_stop", "german_normalization", "german_stemmer"]
        }
      },
      "filter": {
        "german_stop": { "type": "stop", "stopwords": "_german_" },
        "german_stemmer": { "type": "stemmer", "language": "light_german" }
      }
    }
  },
  "mappings": {
    "properties": {
      "content": { "type": "text", "analyzer": "german_content" }
    }
  }
}

8. Testing and debugging with the _analyze API

The _analyze API is the most important tool for understanding an analyzer's behavior before applying it to production data. It accepts either a named analyzer configured on an index, or an ad hoc combination of tokenizer and filters, and returns the position, start offset, end offset and type for every produced token. This makes it visible why a given search term matches or does not match, without actually indexing a test document.

A proven debugging workflow: first run the text in question against the index time analyzer and note the produced tokens, then repeat the same test with the search term against the search time analyzer. If the two token lists do not line up, that is exactly where the cause of an unexpected search result lies. In practice, this comparison saves considerable time compared to trying to solve the problem through query adjustments alone.

# Test a named analyzer configured on an index
curl -s -X GET "https://localhost:9200/articles/_analyze?pretty" \
  -H 'Content-Type: application/json' -u elastic:changeme -d '{
    "analyzer": "german_content",
    "text": "Der Bahnhofsvorplatz wird neu gestaltet"
  }'

# Test an ad-hoc combination without touching the index
curl -s -X GET "https://localhost:9200/_analyze?pretty" \
  -H 'Content-Type: application/json' -u elastic:changeme -d '{
    "tokenizer": "standard",
    "filter": ["lowercase", "asciifolding"],
    "text": "Cafe Muenchen"
  }'

9. Index time and search time analyzers working together

An often overlooked detail: a text field can use two different analyzers, one for indexing and one for searching, configured via the analyzer and search_analyzer parameters in the mapping. By default the same analyzer is used for both phases, which is correct for most cases. But there are legitimate reasons to separate them, for example with synonym expansion: synonyms are often applied only at index time, so that a document containing "laptop" is also found when searching for "notebook", without introducing extra, uncontrolled expansions during the search itself.

A common mistake happens when index time and search time analyzers drift apart unintentionally, for example because a custom analyzer was only updated for one of the two phases. The result is search behavior that works inconsistently: some documents get found, similar, correctly indexed documents do not, without any obvious error appearing in the query. The _analyze API with an explicit analyzer name is the fastest way to expose the discrepancy in such cases, before hunting for the bug in the query logic.

For Magento operators using the Elasticsearch catalog index, it is relevant that the storefront's language selection directly determines which language specific analyzer is used for product names and descriptions. A shop with German and English storeviews typically needs different analyzer configurations per language, otherwise German content gets processed with an analyzer optimized for English, and compound words as well as umlauts get handled suboptimally.

This table summarizes the division of roles: only the tokenizer is strictly present exactly once, character filters and token filters are optional and can be combined as many times as needed. Anyone debugging an analyzer should always first check which tokenizer is used, because it determines the fundamental splitting logic that every subsequent filter builds on.

10. Summary

An analyzer is a fixed pipeline of character filters, exactly one tokenizer, and token filters, that together determine how text is transformed into comparable tokens at index time and at search time. The standard analyzer covers generic cases, but hits limits with language specific requirements like German compounds, which language specific analyzers or a self assembled custom analyzer can solve.

The _analyze API makes an analyzer's otherwise invisible behavior tangible and is the central tool for solving search problems systematically instead of by guessing. Anyone who additionally understands that index time and search time analyzers can drift apart finds the cause of inconsistent search results considerably faster than through trial and error on the query alone.

11. FAQ: Analyzers and Tokenizers in Elasticsearch