Unicode for Web Developers: From Input to Storage and Search

01

Map the boundaries before changing the code

Text crosses several boundaries on its way from a browser to a database and back. Some components handle strings; others handle bytes. An encoding defines how those representations relate. It is not a label that can repair content after a wrong decoder has already changed it. Declaring UTF-8 later does not automatically reverse an earlier misinterpretation.

For a new web page, use a consistent UTF-8 setup and make the response metadata, document declaration, and actual bytes agree. Inspect the real response when diagnosing problems rather than trusting the editor's status bar. For an imported legacy file, establish the source encoding and decide how decoding errors will be reported. Repeatedly trying decoders until something looks plausible is a poor substitute for provenance, especially if the chosen result overwrites the only original.

A small boundary table is useful: what does the input component produce, how does the request represent it, how does the server parse it, what does the database store, and what encoding does the export use? Keep a sample at each step. That turns a system-wide complaint about corrupted text into a precise question about the first boundary at which the content changes. The same table becomes a checklist when another service joins the pipeline.

02

Measure encoded bytes instead of guessing

TextEncoder produces UTF-8 bytes from a JavaScript string, so the resulting byte array's length measures that representation. Do not implement storage limits by counting English letters as one and Chinese characters as three. Real input can include many scripts, combining marks, and characters from other ranges. Let an encoder calculate the byte representation the receiving system will actually use.

Decoding needs an error policy. TextDecoder can replace malformed input by default; its fatal option makes decoding errors throw. A forgiving preview and a strict data import may reasonably choose different behavior. The important questions are whether the caller knows a replacement happened and whether the original bytes remain available. A user cannot correct a damaged import if the application quietly discards the source and presents a replacement-filled string as exact data.

Streaming introduces another boundary. A multibyte character may span two chunks, so decoding each chunk independently can fail even when the complete byte stream is valid. Use decoding that preserves state across chunks and finish the stream correctly. A focused test cuts the same byte sequence at different positions and checks the reconstructed text. This is often more informative than feeding the implementation a very large ASCII file, whose character boundaries happen to coincide with every byte boundary.

03

JavaScript strings are not automatically well formed

JavaScript strings are organized as UTF-16 code units. The length property is not a count of visible characters, and a string can contain an unpaired surrogate. Such a value is not a well-formed sequence of Unicode scalar values. Modern implementations expose isWellFormed and toWellFormed; check support in the environments you target rather than assuming every older runtime has them.

Replacing an unpaired surrogate with a replacement character does not recover the missing information. The replacement operation cannot know what the absent half should have been. A casual text editor may choose explicit tolerance, while a lossless import workflow may need to reject the value and identify the problem. Make that decision at the boundary, before an encoder replaces data and another component reports the result as an exact copy.

Fixed-position slicing is one way to create this problem. Truncating a title by code units can cut through a surrogate pair. Inspecting the boundary and choosing the right segmentation rule preserves more information than repeatedly applying a repair function after the damage. Write the requirement first: are you limiting storage, drawing a short preview, or enforcing a business identifier rule? The convenient string method should follow that decision, not define it by accident.

An empty string and a malformed string are different conditions. A value can be empty and well formed; it can also look innocuous while containing an unpaired surrogate. Required-field checks, size checks, and well-formedness checks answer separate questions. Test them independently so that a successful replacement operation is not mistaken for satisfaction of every business rule.

04

Use the counting unit that matches the interface

Iterating code points avoids splitting a surrogate pair, but it can still separate a base letter from a combining mark or break a compound emoji sequence. User-facing truncation and display counts often need grapheme clusters, which better approximate the units readers perceive. Intl.Segmenter can segment at grapheme granularity; the internationalization data provided by the runtime remains part of the behavior to verify.

If an interface says twenty characters, make clear whether that means displayed units or a storage limit. A product can enforce both a grapheme limit and a UTF-8 byte limit, but those are different constraints. Preserve complete graphemes when shortening a preview. Also account for composition while a user is still entering text; aggressive truncation during that process can interfere with an input method before the intended character is committed.

The example below measures one string in four ways: UTF-16 code units, code points, grapheme clusters, and UTF-8 bytes. Different answers are expected. Each is correct for a different question. Keep the measurement logic separate from the policy deciding which limit matters, then test the policy in the real form. A friendly counter is not enough if the server rejects the same text using an undisclosed unit.

A grapheme count is not a pixel-width measurement. A complete emoji, a Chinese character, and a Latin letter can each form one cluster while occupying different space. If the requirement is to fit a button label, use layout-aware presentation. If the requirement is an input limit, enforce the published counting rule. Keeping these jobs separate avoids using a text statistic as a substitute for typography.

const text = "Ae\u0301😀";
const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });
console.log({
  codeUnits: text.length,
  codePoints: [...text].length,
  graphemes: [...segmenter.segment(text)].length,
  utf8Bytes: new TextEncoder().encode(text).length,
});
// { codeUnits: 5, codePoints: 4, graphemes: 3, utf8Bytes: 8 }

const broken = "\uD800";
console.log(broken.isWellFormed()); // false
console.log(broken.toWellFormed() === "\uFFFD"); // true

try {
  new TextDecoder("utf-8", { fatal: true }).decode(Uint8Array.of(0xFF));
} catch (error) {
  console.log(error.name); // TypeError
}

05

Check the database's own rules

Do not assume that every length in a database type declaration means bytes. PostgreSQL's length-constrained character types use character limits, while other products and surrounding layers may impose different constraints. A value that fits its field can still encounter an index, request-size, or export limit. Read the documentation for the exact product and version, then test the actual path used by the application.

Uniqueness also needs attention. A collation can affect how case, accents, or certain equivalent representations compare. If the application regards two account names as the same but the database accepts both, or the reverse, users encounter confusing conflicts. Design matching keys and database constraints together. Use identical fixtures for the application comparison and the storage constraint so that a disagreement becomes visible before release.

For a migration, inspect existing data under the proposed rules before changing production behavior. Successfully inserting a few multilingual words into a fresh table does not demonstrate that historical records remain distinct. Exercise backup restoration, bulk import, and rollback, and account for older clients still sending older formats. The meaningful test is a documented round trip through the whole storage path, including any transformation that the design intentionally allows.

06

Equality, search, and sorting are different jobs

Exact equality preserves distinctions. Search may intentionally ignore some distinctions. Sorting should often follow language expectations. One general-purpose cleanString function is unlikely to express all three requirements well. A search feature might accept an unaccented query for an accented name, but that convenience does not justify permanently removing the accent from the stored name.

Keeping the original and deriving a search key can make the design easier to explain. The key still needs a specification: normalization, case handling, tokenization, and collation address different issues, and their order can matter. Do not infer global behavior from a few English words. Code point order is a useful deterministic order in some tools, but it is not a universal dictionary order.

Sorting choices also affect pagination. When several values compare equally under the selected collation, a stable secondary key can prevent ambiguous page boundaries. Updates to collation data may require database maintenance, including attention to affected indexes. Record the relevant settings and versions so that a future change can be attributed to new content, new rules, or an implementation defect. Without that record, a harmless ordering change and a serious data problem can look frustratingly similar to the team investigating them.

07

Input and copying belong to the contract

Input methods may update a composition before the user commits the final text. Rewriting or truncating the value on every event can interrupt that process. Treat immediate feedback and final validation as separate concerns. Pasted content deserves the same clarity: explain transformations rather than silently removing every character the implementation did not anticipate.

A copy action should do what its label promises. Copy original preserves the original; copy normalized text identifies the transformation; copy code points exports a diagnostic representation. If the preview and clipboard result differ without explanation, users lose a reliable way to investigate mistakes. A good check compares the copied result with the current tool output at the sequence level instead of merely asserting that a success badge appeared.

Errors should help users act. Where appropriate, identify the position, code point, and field rule rather than saying only invalid character. Preserve the input so it can be corrected. Debugging does not require collecting the user's full text in analytics or shared logs: fixed error categories, bounded counts, and controlled fixtures can establish much of the behavior. The useful evidence is the failed contract and its location, not an unnecessary archive of private messages or credentials.

08

Use a small fixture that crosses real boundaries

Start with ASCII, a combining-accent sequence, Chinese text, a character outside the Basic Multilingual Plane, a complete emoji sequence, and a mixed-direction phrase. Run each through input, encoding, transmission, storage, retrieval, and copying. State which steps must preserve the exact sequence and which may perform a documented transformation. Give each allowed transformation an expected result so that an intentional change is distinguishable from accidental damage.

Include failures: malformed UTF-8, unpaired surrogates, a proposed cut inside a sequence, and input that exceeds the byte limit while remaining below a display-count limit. Vary chunk boundaries for streaming tests and use real database constraints for uniqueness tests. You do not need to enumerate every Unicode character to obtain useful coverage. You need to cover the boundaries and decisions your implementation actually uses.

Connect the fixture to maintenance. Runtime and internationalization-data updates can alter segmentation or sorting behavior, while font changes may alter display without changing storage. Record versions and rerun the relevant examples. When a result changes, investigate whether the requirement, data, or implementation changed before updating the expected value.

Give each fixture a short rationale. A combining sequence exercises segmentation, a supplementary character exercises surrogate handling, and a multibyte sequence exercises decoder boundaries. This makes the suite easier to extend when a real defect appears. A mysterious collection of unusual symbols may still run, but it gives the next maintainer little guidance about what must be preserved during a refactor.

A well-designed text pipeline is explicit about whether each stage handles bytes, code units, code points, graphemes, or language-specific rules. Once those responsibilities are visible, Unicode support becomes a collection of testable contracts rather than a promise that the application accepts a few non-English sample strings.