Configuring Custom Analyzers for Multilingual Shops

1. Why standard analyzers fall short for multilingual shops

Elasticsearch's standard analyzer tokenizes text on word boundaries, lowercases it, and removes no language specific stopwords at all. For a multilingual shop that means a search for "shoe" does not automatically find documents containing "shoes" or "shoed", because without a custom analyzer no stemming takes place. Every language has its own grammatical rules that a universal analyzer cannot represent without making compromises for every one of them.

The problem gets worse for languages with rich morphology such as German, where a single noun can take on many surface forms depending on case, number and compounding. Without a custom analyzer with a German stemmer, a shop would either have to index every word form explicitly or force users to type the exact form used in the product title, which drastically reduces the hit rate of a product search.

Another aspect concerns stopwords: words like "and", "the", "for" in English or "und", "der", "fuer" in German carry almost no meaning for search relevance, but without filtering they unnecessarily bloat the inverted index and dilute scoring. A language specific custom analyzer removes exactly the stopwords that are actually irrelevant for that language, instead of using a blanket, often ill fitting list for every language at once.

2. Structure of a custom analyzer: tokenizer, filter, char filter

A custom analyzer in Elasticsearch consists of three building blocks processed in a fixed order: first optional char_filter steps that transform the raw text before tokenization, then exactly one tokenizer that splits the text into individual tokens, and finally a chain of filter steps that further process each token, for example through lowercasing, stemming or stopword removal.

This component structure makes a custom analyzer fully configurable: instead of adopting one of the predefined language analyzers such as german or french unchanged, individual filter steps can be swapped or extended deliberately, for example to add a custom synonym filter or a project specific normalization.

PUT products_de
{
  "settings": {
    "analysis": {
      "filter": {
        "german_stop": {
          "type": "stop",
          "stopwords": "_german_"
        },
        "german_stemmer": {
          "type": "stemmer",
          "language": "light_german"
        }
      },
      "analyzer": {
        "german_custom": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "german_normalization",
            "german_stop",
            "german_stemmer"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "german_custom" }
    }
  }
}

The german_normalization filter unifies umlauts and the sharp S into their base form before german_stemmer derives the word stem. This order matters: a custom analyzer chain that stems first and normalizes afterward often produces different and worse results, because the stemmer then operates on umlauts that have not been normalized yet.

3. Configuring language specific stemmers correctly

Stemmers reduce words to their word stem so that "run", "runs" and "running" are recognized as related during search. Elasticsearch ships built in stemmers for most European languages, and for German the typical choice is between german, the classic Snowball stemmer, and light_german, a less aggressive variant. The light stemmer reduces less strongly, which favors precision, while the classic stemmer cuts more aggressively and thereby delivers more recall at the cost of occasional false matches.

For French and English the same principle applies with language specific quirks: the French stemmer has to handle elisions such as "l'" and accent marks, while the English stemmer mainly deals with regular plural and verb forms. Choosing the right stemmer inside a custom analyzer is not a purely technical decision, it depends on the product catalog: for technical terminology where exact spelling matters, a lighter stemmer or an additional keyword field without stemming as an exact match fallback is often worthwhile.

PUT products_fr
{
  "settings": {
    "analysis": {
      "filter": {
        "french_elision": {
          "type": "elision",
          "articles_case": true,
          "articles": ["l", "m", "t", "qu", "n", "s", "j", "d", "c", "jusqu", "quoiqu", "lorsqu", "puisqu"]
        },
        "french_stop": { "type": "stop", "stopwords": "_french_" },
        "french_stemmer": { "type": "stemmer", "language": "light_french" }
      },
      "analyzer": {
        "french_custom": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["french_elision", "lowercase", "french_stop", "french_stemmer"]
        }
      }
    }
  }
}

4. Managing stopword lists per language

Elasticsearch ships predefined stopword lists for most languages through shortcuts like _german_, _french_ or _english_, which serve as a starting point inside any custom analyzer. These default lists cover the most common function words, but they are rarely optimal for a specific shop whose search behavior differs from general language usage.

In practice it pays to maintain project specific stopword lists that either extend the defaults or deliberately remove entries. A B2B shop might, for example, remove the word "set" from the stopword list because it carries relevant meaning in product names, while a general word like "new" is deliberately added as a stopword because it appears in nearly every second product title and offers no discriminating power. This adjustment is made through an external file referenced by the custom analyzer.

PUT products_de
{
  "settings": {
    "analysis": {
      "filter": {
        "german_stop_custom": {
          "type": "stop",
          "stopwords_path": "analysis/stopwords_de_custom.txt"
        }
      },
      "analyzer": {
        "german_custom": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "german_normalization", "german_stop_custom", "german_stemmer"]
        }
      }
    }
  }
}

The file stopwords_de_custom.txt lives in the cluster's config directory and can be version controlled in the repository. Changes to the stopword list only take effect after an analyzer cache reload, or more reliably after reindexing the affected documents, since already indexed tokens are not adjusted retroactively.

5. Per-field-per-language: the mapping strategy

The central architectural decision for multilingual shops is whether each language gets its own index or a shared index with language specific fields is used. The per-field-per-language strategy creates a dedicated sub-field for every supported language, for example name.de, name.fr, name.en, each with the matching custom analyzer for that language, while all languages live in the same physical document and index.

PUT products
{
  "settings": {
    "analysis": {
      "analyzer": {
        "de_custom": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "german_normalization", "german_stop", "german_stemmer"] },
        "fr_custom": { "type": "custom", "tokenizer": "standard", "filter": ["french_elision", "lowercase", "french_stop", "french_stemmer"] },
        "en_custom": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "english_stop", "english_stemmer"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "properties": {
          "de": { "type": "text", "analyzer": "de_custom" },
          "fr": { "type": "text", "analyzer": "fr_custom" },
          "en": { "type": "text", "analyzer": "en_custom" }
        }
      }
    }
  }
}

This structure allows a single query to target exactly the right language field without managing or syncing several indices. The downside compared to separate indices per language is a somewhat larger document size, because every document carries all language versions, which can matter for very large catalogs with many languages. For most multilingual shops, however, the operational simplicity of a shared index clearly outweighs that, especially because updating a product only requires a single write instead of one per language index.

6. Maintaining synonyms per language

Synonym filters extend a custom analyzer with the ability to unify different terms for the same concept, for example "sofa" and "couch" in English. Because synonym pairs are language specific and rarely translate sensibly, each language must get its own synonym list, closely tuned to the vocabulary and search habits of its target audience.

Technically, the synonym filter is added as another step in the filter chain of the language specific custom analyzer, typically after lowercasing and before stemming, so synonyms match on normalized but not yet stemmed tokens. When changing the synonym list, keep in mind that an index-time synonym filter only affects newly indexed documents, while a synonym_graph filter applied at search time takes effect immediately without reindexing, though with different performance characteristics for complex multi-word synonyms.

7. Umlauts, diacritics and ICU normalization

For languages with diacritics, such as German umlauts, French accents or Scandinavian special characters, the simple asciifolding filter is often not enough, because it reduces every special character to its ASCII base form indiscriminately, blurring semantically relevant differences. The ICU analysis plugin offers icu_folding as a more language aware normalization that distinguishes between characters that are truly equivalent and those that carry their own meaning.

For a German custom analyzer, the built in german_normalization filter is usually the more precise choice, because it is tailored specifically to German umlaut rules, for example converting "ae" to "a" only in certain contexts, while icu_folding serves as a generic, cross language solution for mixed catalogs with many different alphabets, for example international marketplaces with Cyrillic or Greek product names.

PUT products_intl
{
  "settings": {
    "analysis": {
      "analyzer": {
        "icu_custom": {
          "type": "custom",
          "tokenizer": "icu_tokenizer",
          "filter": ["icu_folding", "lowercase"]
        }
      }
    }
  }
}

8. Testing a custom analyzer with the _analyze API

Before a custom analyzer goes to production, it should be tested against typical search terms through the _analyze API. The endpoint shows exactly which tokens a given text produces after every filter step, immediately revealing whether a stemmer cuts too aggressively, whether a stopword list removes a relevant word, or whether a synonym does not match as expected.

# Test how the german_custom analyzer tokenizes a search term
curl -s -X POST "https://es.mironsoft.de:9200/products_de/_analyze" \
  -H "Content-Type: application/json" -d '{
    "analyzer": "german_custom",
    "text": "Laufschuhe fuer Damen"
  }' | jq '.tokens[].token'

# Expected output roughly: lauf, schuh, dame

A systematic testing approach for every custom analyzer includes a list of real search queries pulled from the logs, run regularly against new analyzer versions before they go to production. That surfaces regressions, for example when a change to the stopword list suddenly removes a word that was actually relevant for a specific product category.

9. Steering cross language search at query time

Even with a clean per-field-per-language structure, the question remains how a search request gets routed to the correct language specific custom analyzer at runtime. The usual approach is to determine the user's language from session context, the store view, or the Accept-Language header, and to target the query deliberately at the matching language field, for example name.de for German speaking users.

For cases where the user's language is unclear, or a user deliberately searches across several languages at once, a multi_match query can target all language fields simultaneously, with each field still analyzed by its own custom analyzer. Boosting individual language fields, for example weighting the main store language higher, ensures that hits in the preferred language appear before hits in other languages, without excluding them entirely.

10. Summary

A custom analyzer per language is the foundation of any precise multilingual search, because stemmers, stopword lists and normalization rules work fundamentally differently between languages. The per-field-per-language strategy with dedicated sub-fields per language inside the same index combines operational simplicity with linguistic precision, while the _analyze API makes every analyzer testable against real search terms before it goes to production.

Anyone who consistently maintains stopword lists and synonyms per language instead of using a blanket configuration for every language, and who deliberately chooses ICU or language specific normalization filters depending on the catalog structure, builds a search that actually delivers the results users expect in every supported language, instead of forcing a compromised one-size-fits-all solution onto every locale at once.

11. FAQ: Custom Analyzers for Multilingual Shops