String.isWellFormed() and toWellFormed(): Handling Broken UTF-16 Strings Safely

1. How JavaScript strings can be broken internally

JavaScript strings are internally stored as sequences of UTF-16 code units, not as sequences of Unicode code points. For characters outside the Basic Multilingual Plane, such as most emoji, rare Chinese characters, or historical writing systems, two code units are needed, a so called surrogate pair: a high surrogate unit followed by a low surrogate unit. As long as both halves stay together, everything works smoothly.

The problem arises when one of these two halves ends up isolated in the string, a so called lone surrogate. This can happen when a string is cut at an arbitrary code unit position, for example during a naive substring() operation right in the middle of a surrogate pair, or when malformed, non standard data from an external source ends up directly in JavaScript. A string with a lone surrogate is technically valid JavaScript, but does not correspond to any validly UTF-16 encoded Unicode text, this is called 'not well formed'.

2. Why lone surrogates become a real problem

As long as such a broken string stays within JavaScript, the problem often goes unnoticed, console.log() usually shows a replacement character and processing appears to continue normally. It becomes critical only once the string crosses a boundary into a different encoding: encodeURIComponent() still works, but TextEncoder.prototype.encode(), which converts UTF-16 JavaScript strings to UTF-8 bytes, either throws an exception or silently replaces the lone surrogate with the Unicode replacement character U+FFFD, depending on context.

For databases this is especially treacherous: MySQL, for instance, expects valid UTF-8 with the utf8mb4 collation, a lone surrogate there frequently causes a hard insert failure or, worse, silently corrupted data that only surfaces weeks later when reading it back. Similarly critical are JSON serialization across network boundaries, file uploads with filenames containing lone surrogates, or passing user input on to APIs that do not themselves use UTF-16.

3. Typical sources of lone surrogates

The most common source is URL parameters decoded via decodeURIComponent(): a manipulated or simply malformed URL can contain a percent encoded sequence that, after decoding, results in an isolated surrogate. Since decodeURIComponent() itself does not perform any well formedness check, the broken result ends up unnoticed further down the program flow.

A second common source is file uploads, particularly filenames from users created on systems with a different character encoding, as well as directly reading foreign binary data that gets mistakenly interpreted as UTF-16 text. Truncating a string after a fixed number of code units, for example a naive character limit for comment fields ('280 characters max'), can also accidentally cut right through a surrogate pair, breaking a string that was previously completely valid.

// A valid emoji consists of a surrogate pair
const emoji = "????"; // high + low surrogate
console.log(emoji.length); // 2 (two UTF-16 code units)

// Naive truncation can tear the pair apart
const broken = emoji.slice(0, 1);
console.log(broken.length); // 1 -- only the high surrogate half

// URL decoding can also produce broken strings
const suspiciousUrl = "%ED%A0%BD"; // invalid UTF-8 bytes for a surrogate
try {
  const decoded = decodeURIComponent(suspiciousUrl);
  console.log(decoded.length);
} catch (e) {
  console.log("decodeURIComponent often throws directly here");
}

4. isWellFormed(): checking without changing

String.prototype.isWellFormed() returns true if the string consists exclusively of complete characters, meaning either single code units outside the surrogate range or complete high low surrogate pairs, and false as soon as at least one lone surrogate is present. The method is a pure check with no side effect, the string itself remains unchanged.

The practical benefit lies in early validation at system boundaries: before a string is passed on to TextEncoder, a database layer, a file API, or an outgoing HTTP request, isWellFormed() lets you check whether special handling is needed, instead of blindly hoping the downstream API will somehow deal with it. For security sensitive code, such as input validation on a public API, this is a simple, fast first filter.

const valid = "Hello ???? World";
console.log(valid.isWellFormed()); // true

const lone = "Hello " + String.fromCharCode(0xD83D); // only a high surrogate
console.log(lone.isWellFormed()); // false

function validateInput(text) {
  if (!text.isWellFormed()) {
    throw new Error("Input contains invalid character encoding");
  }
  return text;
}

5. toWellFormed(): repairing instead of rejecting

While isWellFormed() only informs, toWellFormed() actively repairs: every lone surrogate is replaced with the Unicode replacement character U+FFFD, all complete characters remain unchanged. The result is guaranteed to be well formed and can be safely passed on to any downstream API without that API needing to implement its own special handling for lone surrogates.

This choice between rejecting (via an isWellFormed() check plus an exception) and repairing (via toWellFormed()) is a deliberate design decision that depends on the specific use case. For security sensitive input, such as a password field or an API signature, rejecting is usually the right choice, since a silently altered password would lead to confusing downstream errors. For user friendly display or storage cases, such as a comment field, repairing is often the better user experience, since the user is not confronted with a cryptic 'invalid character encoding' error.

const brokenString = "Comment " + String.fromCharCode(0xD83D) + " end";
console.log(brokenString.isWellFormed()); // false

const repaired = brokenString.toWellFormed();
console.log(repaired.isWellFormed()); // true
console.log(repaired); // "Comment \uFFFD end"

// Now safe for TextEncoder/UTF-8 conversion
const encoder = new TextEncoder();
const bytes = encoder.encode(repaired); // guaranteed to work without exception

6. Practical example: safeguarding at API boundaries

A typical usage pattern is an Express or Fastify middleware layer that checks every incoming request body for well formedness before further processing. Instead of having to safeguard every single string processing function in the backend against lone surrogates, a central check right at the entry point suffices, matching the principle of 'validate at the system boundary' and freeing the rest of the code from this special handling.

This is similarly relevant for file upload handlers extracting filenames from the Content-Disposition header or from multipart form data. These filenames originate from user controlled input and can indeed be malformed, especially for uploads from older or exotic client systems. A toWellFormed() call before storing a filename in a file system or database reliably prevents a single broken filename from crashing the entire upload process.

function middleware(req, res, next) {
  for (const [key, value] of Object.entries(req.body)) {
    if (typeof value === "string" && !value.isWellFormed()) {
      req.body[key] = value.toWellFormed();
    }
  }
  next();
}

// File upload: sanitize filename before storing
function storagePath(filename) {
  const safeName = filename.toWellFormed();
  return `/uploads/${safeName}`;
}

7. The effort required before these methods existed

Before isWellFormed() and toWellFormed(), such a check had to be implemented manually via a regular expression, for example a pattern specifically searching for isolated high or low surrogates not directly followed by, or not directly preceded by, their respective partner. Such a pattern is error prone, hard to read, and had to be reimplemented in every project or imported from a third party library.

Alternatively, many projects resorted to a workaround via TextEncoder/TextDecoder with the fatal flag to detect broken strings indirectly through an exception, which however caused unnecessary overhead from a full byte conversion just to perform a plain yes no check. The native methods, by contrast, are both semantically clearer and more performant, since the engine can run the check directly on the internal UTF-16 representation without a detour through byte conversion.

8. Browser support and polyfill recommendation

isWellFormed() and toWellFormed() are part of ES2024 and are natively supported by Chrome and Edge since version 111, Firefox since version 119, Safari since version 17, and Node.js from version 20 onward. For projects that need to support older runtime environments, core-js offers a complete polyfill implementation of both methods that behaves transparently.

Since both methods are comparatively new, it is worth adding a feature detect in production code before relying on them, especially in environments with mixed browser support such as Electron apps with an older embedded Chromium or legacy server runtimes. A simple check like typeof "".isWellFormed === "function" before loading a polyfill saves unnecessary load time in modern environments.

9. Comparison table: validation strategies at a glance

The table below puts the native solution next to previous workarounds and shows when which approach makes sense, to make the decision easier for your own use case.

As a rule of thumb: for new projects with a modern runtime target, the native methods are always the first choice, since they are both more readable and more performant than any manual alternative. For existing legacy codebases with older target environments, a polyfill or the regex based solution remains the pragmatic transitional option.

10. Summary

11. FAQ: isWellFormed() and toWellFormed() at a glance