Configuring Custom OpenSearch Analyzers for Magento Product Search

1. Where the default analyzer falls short

Magento's default analyzer combines a standard tokenizer with a lowercase filter, ASCII folding and a snowball stemmer. That combination works well for ordinary product names and description text, but for SKUs containing hyphens, slashes or dots it often produces unusable tokens, because the tokenizer treats those characters as word boundaries.

A second, frequently underestimated problem concerns compound German words. A search for Gartenschlauch often fails to find a product titled Gartenschlauchhalter, because the default analyzer treats the compound as a single, indivisible token and performs no language specific decomposition. These two cases, special characters in SKUs and German compounds, are exactly what this article focuses on.

This is deliberately scoped away from general relevance tuning and synonyms, which are covered elsewhere. The focus here sits purely at the analyzer and tokenizer level, meaning how text is broken into searchable tokens in the first place, before scoring or synonyms even come into play.

2. How Magento builds mapping and analyzers for the search index

Magento does not rebuild the index settings for catalogsearch_fulltext by hand on every indexer run. Instead it reads a declarative configuration shipped with the Magento_Elasticsearch module that describes tokenizers, filters and analyzer definitions. That configuration is sent to OpenSearch as a settings block whenever a new index is created, alongside the actual field mapping for attributes such as sku, name or description.

The important part for custom extensions is that this analyzer configuration is modular and merged through the usual Magento mechanisms, much like db_schema.xml or events.xml. A custom module can therefore declare additional analyzers, filters and char filters without touching the core Elasticsearch module code.

One thing to keep in mind: a change to the analyzer configuration only takes effect after a full reindex, since OpenSearch cannot alter analyzer settings on an existing index after the fact. Magento already creates a new, versioned index on every reindex run and only switches the alias once that run completes successfully, which fits analyzer changes very well.

3. Registering a custom analyzer through a module

A custom analyzer gets registered through a lightweight module that only depends on Magento_Elasticsearch and needs no blocks or controllers of its own. The declaration lives in a dedicated es_indexer.xml that defines char_filter, filter and analyzer nodes, which get merged with the core definitions into a single settings object during module merging.

The analyzer name must be unique project wide, so a vendor prefix such as mironsoft_sku_analyzer is worth adopting to avoid collisions with future core analyzers or other third party modules. After a reindex, the new analyzer can be addressed exactly like any built in analyzer through the OpenSearch API.

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Elasticsearch:etc/es_indexer.xsd">
    <char_filter name="mironsoft_sku_char_filter" type="pattern_replace">
        <pattern><![CDATA[[\/\.]]]></pattern>
        <replacement><![CDATA[_]]></replacement>
    </char_filter>
    <filter name="mironsoft_german_decompounder" type="dictionary_decompounder">
        <word_list_path><![CDATA[analysis/german_compounds.txt]]></word_list_path>
        <min_subword_size><![CDATA[4]]></min_subword_size>
    </filter>
    <analyzer name="mironsoft_sku_analyzer">
        <char_filter><![CDATA[mironsoft_sku_char_filter]]></char_filter>
        <filter><![CDATA[lowercase]]></filter>
        <tokenizer><![CDATA[keyword]]></tokenizer>
    </analyzer>
    <analyzer name="mironsoft_german_analyzer">
        <filter><![CDATA[lowercase,mironsoft_german_decompounder,german_normalization]]></filter>
        <tokenizer><![CDATA[standard]]></tokenizer>
    </analyzer>
</config>

4. Tokenizing SKUs with special characters cleanly

For sku, a keyword tokenizer usually applies, treating the entire value as a single token, combined with a pattern_replace filter that replaces separators such as slashes or dots with underscores. That keeps the SKU exactly matchable while different spellings such as ABC-123 and ABC/123 map to the same normalized token.

Anyone wanting to allow substring search on SKUs as well, for example finding ABC-123 when a customer types 123, needs a second, edge_ngram based analyzer on a separate search field, since a pure keyword analyzer only ever returns exact matches. Both analyzers can be attached to the same attribute in parallel through a multi field mapping.

// OpenSearch _analyze call to test the SKU analyzer
POST /magento2_default_catalogsearch_fulltext/_analyze
{
  "analyzer": "mironsoft_sku_analyzer",
  "text": "ABC-123/DE.v2"
}

// Expected output (trimmed)
{
  "tokens": [
    { "token": "abc_123_de_v2", "start_offset": 0, "end_offset": 13, "type": "word" }
  ]
}

5. Splitting German compound words with the dictionary decompounder

The dictionary_decompounder filter needs a maintained word list stored in the index analysis directory, from which it recognizes subwords. For Gartenschlauchhalter, with garten, schlauch and halter present in that list, the filter emits the three subword tokens in addition to the original token, so a search for Gartenschlauch still finds the product.

That word list is the most maintenance heavy part of this approach, since OpenSearch does not ship automatic morphological decomposition for German the way Hunspell dictionaries do for some other languages. In practice, a category specific list of the two hundred to four hundred most common terms in a given assortment works well, maintained as a text file in the analysis directory and kept synchronized across all store environments through a deploy script.

Important: the decompounder analyzer should only be applied to name and dedicated search fields, not to sku, since splitting SKUs into subword tokens would produce false matches. Field assignment is therefore handled per attribute, as shown in the next section.

# analysis/german_compounds.txt (excerpt), lives in the OpenSearch data path analysis directory
garten
schlauch
halter
werkzeug
koffer
akku
schrauber

# The file must be identical on every OpenSearch node, otherwise one
# shard returns different tokens than another.

6. Assigning analyzers to specific fields

Mapping an analyzer to a concrete attribute happens through the field mapping that Magento builds alongside the analyzer definitions. A plugin on the responsible field provider makes it possible to override the analyzer or search_analyzer value in the generated mapping for selected attribute codes, instead of applying the global default analyzer to every text field.

It is worth distinguishing between the index analyzer and the search analyzer, since the two do not have to match. For sku, using the same analyzer in both directions makes sense, while for name it can help to apply the decompounder only at index time, and use a leaner analyzer without decomposition for the actual search query, to avoid ambiguity on very short search terms.

<?php

declare(strict_types=1);

namespace Mironsoft\SearchAnalyzer\Plugin;

use Magento\Elasticsearch\Model\Adapter\FieldMapper\Product\FieldProvider\Base\Field\FieldTypeConverter;

/**
 * Assigns a custom OpenSearch analyzer to selected product attributes,
 * instead of applying the global default analyzer to every text field.
 */
class AssignCustomAnalyzerPlugin
{
    /**
     * Attribute code to analyzer name, only overrides the explicitly listed fields.
     *
     * @var array<string, string>
     */
    private const FIELD_ANALYZER_MAP = [
        'sku' => 'mironsoft_sku_analyzer',
        'name' => 'mironsoft_german_analyzer',
    ];

    /**
     * Overrides the analyzer entry in the generated field mapping for the listed attributes.
     *
     * @param FieldTypeConverter $subject
     * @param array<string, mixed> $result
     * @param string $attributeCode
     * @return array<string, mixed>
     */
    public function afterConvert(FieldTypeConverter $subject, array $result, string $attributeCode): array
    {
        if (isset(self::FIELD_ANALYZER_MAP[$attributeCode])) {
            $result['analyzer'] = self::FIELD_ANALYZER_MAP[$attributeCode];
        }

        return $result;
    }
}

7. Reindexing and verifying analyzer behavior through the API

Every change to es_indexer.xml requires a full reindex of the full text index, because analyzer settings are only applied when a new index gets created. A simple cache flush or a partial reindex is not enough, which is why these changes should always be tested in a staging environment first.

The fastest way to verify actual tokenization is not through the Magento search box but directly through the OpenSearch _analyze API. It shows exactly which tokens a given analyzer produces for a given piece of text, independent of whether and how a product actually gets found later on.

Only once the tokens look correct at the API level does it make sense to move on to searching for a real product in the storefront. This order saves considerable time in practice, since analyzer misconfigurations otherwise only surface late, and in a hard to trace form, inside the search results themselves.

# Reindex the search index after changing es_indexer.xml
bin/magento indexer:reindex catalogsearch_fulltext

# Verify analyzer behavior directly against the live index
curl -s -X POST "https://opensearch:9200/magento2_default_catalogsearch_fulltext/_analyze" \
  -H "Content-Type: application/json" \
  -d '{"analyzer": "mironsoft_german_analyzer", "text": "Gartenschlauchhalter"}' | jq .

8. How this differs from relevance tuning and synonyms

Analyzer configuration and relevance tuning solve different problems and should not be conflated. The analyzer decides which tokens even exist, and therefore whether a document is a candidate for a given query at all. Boosting, field weighting and function scores only decide, after that, in what order the matched candidates get displayed.

Synonyms behave similarly, sitting as their own filter inside the analyzer chain, but representing a separate maintenance problem, since synonym lists are usually maintained by domain rather than by language. A decompounder filter for compounds does not replace a synonym list, and neither does the reverse. Both mechanisms complement each other but should be tested independently, to keep cause and effect cleanly separated when a search problem appears.

9. Pitfalls in production

In multi store setups with several languages, the German analyzer must not apply globally to every store view, or it will degrade tokenization of English or French product data. Field assignment in the field provider plugin therefore needs to be store or language aware, usually by checking the current store code within the context of the reindex run.

A second pitfall concerns the decompounder word list, which needs to reach every OpenSearch node in a cluster in sync on every deploy. If the file differs by even a single line between nodes, identical queries return different results depending on which shard gets hit, a symptom that is hard to reproduce unless the root cause is already known.

Finally, every analyzer change should first be tested against a copy of the production index in staging, including a full reindex, before going live. Because switching analyzers always forces a complete index rebuild, untested changes in production come with noticeable system load and, in the worst case, a briefly incomplete search experience.

10. Summary

11. FAQ: OpenSearch analyzers