Intl.Segmenter: Correctly Splitting Text into Words, Sentences, and Grapheme Clusters

1. Why String.split() is not enough for text segmentation

The obvious solution for splitting a text into words is text.split(' '). For simple English or German prose this works on the surface, but it immediately fails on several real world requirements: punctuation sticks to the word ('Hello,' instead of 'Hello'), multiple spaces produce empty entries, and most importantly: many of the world's languages do not separate words with spaces at all. Chinese, Japanese, and Thai write words directly next to each other, a space split there simply returns the entire sentence as a single 'word'.

The problem is even more subtle for character by character iteration using [...text] or a for...of loop. This correctly respects surrogate pairs and does not break emoji like ???? into broken halves, but it fails on composed emoji like family emoji with zero width joiners (????‍????‍????‍????) or on letters with combining accent marks: what a human perceives as a single visual character, a so called grapheme cluster, can consist of several Unicode code points, which naive iteration incorrectly counts as separate 'characters'.

2. Basics: instantiating and using Intl.Segmenter

Intl.Segmenter is part of the ECMA-402 internationalization APIs and is instantiated with a locale string and an options object that uses the granularity option to determine the level at which to segment: 'grapheme' for visual character clusters, 'word' for words, or 'sentence' for sentences. The .segment(text) method then returns an iterable segments object whose individual entries contain, besides the actual text segment, metadata such as the start position and, for word segmentation, an isWordLike flag that distinguishes actual words from pure separators like spaces or punctuation.

The decisive strength compared to a hand rolled solution is that Intl.Segmenter implements the full Unicode Text Segmentation Algorithm (UAX #29), a cross language rule set specified by the Unicode Consortium and continuously updated with new Unicode versions. Implementing these rules in your own code would require hundreds of edge cases, most of which few developers even know exist.

const segmenter = new Intl.Segmenter("en", { granularity: "word" });
const text = "Hello, how are you?";

for (const { segment, isWordLike } of segmenter.segment(text)) {
  console.log(`"${segment}" -- word: ${isWordLike}`);
}
// "Hello" -- word: true
// "," -- word: false
// " " -- word: false
// "how" -- word: true
// ...

// Filter out only actual words
const wordsOnly = [...segmenter.segment(text)]
  .filter(s => s.isWordLike)
  .map(s => s.segment);
console.log(wordsOnly); // ["Hello", "how", "are", "you"]

3. Grapheme clusters: counting emoji and accents correctly

With granularity: 'grapheme', Intl.Segmenter splits text into exactly the units a human intuitively perceives as 'one character'. A composed family emoji, technically consisting of four individual person emoji and three zero width joiners, is correctly recognized as a single grapheme cluster, while a naive [...text] iteration would incorrectly break it into seven separate 'characters'. The same applies to letters with combining diacritical marks, for instance an 'e' followed by a standalone acute accent code point instead of a precomposed 'é'.

This correct character counting is not an academic subtlety, it has direct practical consequences: a character limit for a comment field computed via text.length or naive code point iteration might count a single complex emoji as five or seven characters, producing a limit that is completely incomprehensible to the user. With grapheme segmentation, the counted number exactly matches what the user sees on screen.

const family = "????‍????‍????‍????"; // one visual character, technically 7 code points

console.log([...family].length); // 7 -- naive iteration counts wrong
console.log(family.length);       // 11 -- .length counts UTF-16 units, even worse

const graphemeSegmenter = new Intl.Segmenter("en", { granularity: "grapheme" });
const graphemes = [...graphemeSegmenter.segment(family)];
console.log(graphemes.length); // 1 -- correct: ONE grapheme cluster

// Practical use for a correct character limit
function correctCharacterCount(text) {
  return [...graphemeSegmenter.segment(text)].length;
}
console.log(correctCharacterCount("Hello ????‍????‍????‍????!")); // 8 instead of 17

4. Word boundaries in CJK languages without spaces

Chinese and Japanese write connected text entirely without spaces between words, a space split produces no meaningful segmentation at all for these languages. Intl.Segmenter solves this problem using a dictionary based algorithm that, for the given locale, determines the most likely word boundaries within the connected character sequence, based on language specific frequency and rule models stored in the ICU (International Components for Unicode) dataset that most JavaScript engines are built on.

For applications with text search, autocomplete, or text wrapping in displays with Chinese or Japanese content, this is essential: without correct word boundary detection, a search function could not distinguish between two adjacent but semantically independent words, and automatic line wrapping would break right in the middle of a coherent word, something native readers would immediately notice as an error.

const chineseText = "我喜欢学习编程语言";

// A space split does not work here at all
console.log(chineseText.split(" ")); // ["我喜欢学习编程语言"] -- a single "word"

const zhSegmenter = new Intl.Segmenter("zh", { granularity: "word" });
const words = [...zhSegmenter.segment(chineseText)]
  .filter(s => s.isWordLike)
  .map(s => s.segment);

console.log(words); // ["我", "喜欢", "学习", "编程", "语言"]
// I, like, learn/study, programming, language -- correctly split into words

5. Sentence segmentation: more than just splitting at periods

With granularity: 'sentence', Intl.Segmenter splits text into individual sentences, and does so significantly more robustly than a naive split on the period character. A simple text.split('.') approach breaks on abbreviations like 'e.g.', decimal numbers like '3.14', or titles like 'Dr.', since these contain periods without marking the end of a sentence. Intl.Segmenter's sentence segmentation accounts for exactly these language specific exceptions via the full UAX-29 rule set.

This is practically relevant for text preview snippets meant to show only the first complete sentence, for language learning applications that split text sentence by sentence for practice, or for text to speech preprocessing, where each sentence is passed individually to a speech synthesis API and an incorrect sentence boundary would produce an unnatural pause mid sentence.

const text = "Dr. Smith arrived at 9.30 a.m. He brought 3.5 kg of documents. Everything was fine.";

const sentenceSegmenter = new Intl.Segmenter("en", { granularity: "sentence" });
const sentences = [...sentenceSegmenter.segment(text)].map(s => s.segment.trim());

console.log(sentences);
// ["Dr. Smith arrived at 9.30 a.m.", "He brought 3.5 kg of documents.", "Everything was fine."]
// -- "Dr." and "9.30" were correctly NOT recognized as sentence endings

6. Direct comparison: Intl.Segmenter versus String.split()

The difference between the two approaches is most clearly shown with a text combining several problem cases at once: emoji, a CJK word without surrounding spaces, and an abbreviation with a period. While split(' ') only splits at actually present spaces and ignores everything else, Intl.Segmenter delivers linguistically meaningful boundaries in every case, regardless of whether the given language uses spaces for word separation at all.

Another practical difference: split() discards the separators themselves, while Intl.Segmenter with word segmentation also returns the separators (spaces, punctuation) as their own segments with isWordLike: false. This allows a text to be segmented completely losslessly and, if needed, for example for syntax highlighting or text markup, reassembled exactly into its original form.

7. Performance considerations and practical usage patterns

Instantiating an Intl.Segmenter object is relatively expensive compared to the actual segmenting, since locale data is loaded and rule sets are prepared. For repeated segmentation with the same locale and the same granularity, a single segmenter instance should therefore always be reused instead of creating a new segmenter on every call, quite similar to other Intl APIs such as Intl.NumberFormat or Intl.DateTimeFormat.

For extremely performance critical use cases with very high throughput, such as real time text analysis over millions of characters per second, it is worth benchmarking against specialized native libraries. For the vast majority of typical use cases, form validation, text preview, character limit calculation, search indexing, the native performance of Intl.Segmenter is more than sufficient and completely avoids the extra bundle size of an external library.

// Create the segmenter once and reuse it
const wordSegmenter = new Intl.Segmenter(undefined, { granularity: "word" });

function countWords(text) {
  let count = 0;
  for (const { isWordLike } of wordSegmenter.segment(text)) {
    if (isWordLike) count++;
  }
  return count;
}

console.log(countWords("This is a test sentence, right?")); // 6

8. Browser support and comparison to the status quo

Intl.Segmenter is natively supported by all current versions of Chrome, Edge, Safari, and Node.js from version 16 onward, Firefox caught up with version 125. For projects that still need to serve older Firefox versions or very old Node LTS releases, JavaScript implementations such as intl-segmenter-polyfill exist, replicating the same API surface with custom generated Unicode rule tables, though with noticeably larger bundle size than the native solution.

Before Intl.Segmenter, projects needing correct grapheme counting or CJK word segmentation had to rely on external libraries such as grapheme-splitter or segmentit, which added extra bundle weight and often only covered a subset of the full UAX-29 specification. The native solution covers all three granularities (grapheme, word, sentence) in a single API already present in every modern browser.

9. Reference table: granularities compared

The table below summarizes the three granularity levels of Intl.Segmenter and matches each with a typical use case, to make it easier to pick the right option for a given project.

As a rule of thumb: use 'grapheme' whenever correct counting or cursor movement within text is involved, 'word' for search indexing and text analysis especially with multilingual content, and 'sentence' for preview snippets, speech synthesis preprocessing, or learning material preparation.

10. Summary

11. FAQ: Intl.Segmenter at a glance