Start with the unit, not the label
A code point is a position in Unicode's code space, conventionally written as U+ followed by hexadecimal digits. A is U+0041, 中 is U+4E2D, and 😀 is U+1F600. That notation identifies a value. It does not, by itself, tell you which bytes appear in a file containing the character.
An encoding form represents scalar values using code units. UTF-8 uses 8-bit units, UTF-16 uses 16-bit units, and UTF-32 uses 32-bit units. When a unit spans several bytes, serializing it also requires a byte order. The number in the encoding's name describes its basic unit, not the number of bits occupied by everything a reader perceives as one character.
This distinction prevents several misleading comparisons. A database limit, a string index, and a caption's visible length may all be numbers, but they need not count the same things. Before comparing implementations, write the unit next to each measurement. “This title is twelve characters long” is too vague if one system means bytes and another means grapheme clusters. Neither can enforce the other's limit safely without doing the appropriate conversion or segmentation.
UTF-8 uses only as many bytes as the scalar value needs
Modern UTF-8 represents a Unicode scalar value in one to four bytes and preserves the single-byte representation of ASCII. Our examples encode as 41 for A, E4 B8 AD for 中, and F0 9F 98 80 for 😀. Those are hexadecimal byte values. Each example contains one code point, yet their encoded sizes are one, three, and four bytes.
The example below makes this visible with TextEncoder. It measures UTF-8 output, not font width or the number of shapes on screen. Changing the typeface will not change these bytes. Changing the underlying character sequence might, even if the result looks similar. Inspecting the actual output is therefore more dependable than estimating size from appearance.
A practical consequence is that an arbitrary byte boundary may fall inside an encoded value. Keeping the first two bytes of a three-byte sequence does not produce a small but usable part of a character. It produces incomplete input. If a protocol imposes a byte budget, use an implementation that respects encoding boundaries, and separately decide whether the text must also end at a complete grapheme boundary.
for (const text of ['A', '中', '😀']) {
const bytes = new TextEncoder().encode(text);
console.log(text, bytes.length,
[...bytes].map(x => x.toString(16).padStart(2, '0')).join(' '));
}
// A 1 41
// 中 3 e4 b8 ad
// 😀 4 f0 9f 98 80UTF-16 sometimes needs a pair of code units
UTF-16 uses one 16-bit code unit for scalar values in the Basic Multilingual Plane and two for supplementary values. The two units representing 😀 are D83D and DE00. Together they encode U+1F600. A surrogate value taken by itself is not that emoji, and surrogate code points are excluded from the set of Unicode scalar values.
This is why JavaScript reports a length of two for 😀 but one for 中. Its string elements and indices are defined in 16-bit units. The result is consistent with that contract. Trouble starts when an application labels the result “characters remaining,” or passes a reader's desired character count straight into a method that slices by code-unit offsets.
Audit the purpose of each length before changing code. An offset returned by a JavaScript API may correctly need code units. A preview may need grapheme clusters. A request budget may need encoded bytes. Give those quantities different names and keep conversions explicit. Replacing every use of length with one alternative is unlikely to be correct, because the application probably has several distinct requirements hiding behind the same word.
UTF-32 fixes code-point width, not visual-character width
UTF-32 uses a single 32-bit code unit for each Unicode scalar value. This simplifies some code-point-oriented operations, but it does not make every 32-bit integer valid text. Values beyond the Unicode range and surrogate values remain excluded. A larger container does not remove the restrictions on what the container is allowed to represent.
It also does not solve grapheme segmentation. An e followed by a combining acute accent still contains two code points when represented in UTF-32. The fact that a reader sees an accented letter does not merge those values into one unit. A program that needs to delete, truncate, or count complete perceived characters still has another layer of work to do.
The size table describes our samples before adding byte-order marks or any external framing. It is useful for understanding the representations, not predicting an application's performance. A real page might combine English markup, Chinese prose, and emoji. Compression, runtime storage, and processing patterns also matter. Measure a representative workload before making an architectural choice from a handful of individual characters.
| Sample | UTF-8 bytes | UTF-16 code units | UTF-32 code units |
|---|---|---|---|
| A | 1 | 1 | 1 |
| 中 | 3 | 1 | 1 |
| 😀 | 4 | 2 | 1 |
| e + combining acute accent | 3 | 2 | 2 |
Byte order tells the reader which part comes first
A UTF-16 code unit occupies two bytes, and a UTF-32 code unit occupies four. A byte stream therefore needs an ordering convention. Big-endian places the more significant part first; little-endian places it last. For 中, UTF-16BE produces 4E 2D, while UTF-16LE produces 2D 4E. The code point has not changed; the serialized arrangement has.
UTF-8 has no comparable endianness choice because each code unit is already a single byte. Reversing its bytes to match a machine's native integer layout would corrupt the encoding. Likewise, swapping the two bytes of each UTF-16 unit is a byte-order conversion, not a way to reverse the text. Similar-looking operations can belong to completely different layers.
An interface specification should name the exact representation it accepts and produces. “Unicode text” does not settle whether a byte stream is UTF-8 or UTF-16LE, whether a signature is expected, or how invalid input is handled. Clear labels are especially valuable at the boundary with legacy files and external tools, where neither side should have to infer those choices from a sample that happens to contain only English.
A BOM is a signal with format-specific rules
A byte-order mark uses U+FEFF at the beginning of certain text streams to signal byte order. UTF-8 does not need byte-order selection, but it can also carry the EF BB BF signature. Whether that signature should be emitted, recognized, or rejected depends on the surrounding format and protocol. Adding one to every file is not a general-purpose compatibility fix.
A useful debugging example is a CSV importer whose first column name appears correct but does not match the expected string. The importer may have preserved a leading marker as content. That calls for investigating how the opening bytes were read, rather than stripping every invisible character from every field. A narrowly scoped correction is easier to justify and test.
Keep the original file while investigating. Inspect its initial bytes and compare them with the format's requirements and the decoder's behavior. By the time an application receives a string, a lower layer may already have consumed the marker. Its absence from the string does not prove it was absent from the file. Conversely, finding U+FEFF inside text requires attention to its position and context.
Error handling is part of the encoding contract
Well-formed Unicode text can be transcoded between these representations while preserving its scalar-value sequence. Real inputs are not always well formed. A file can be truncated, a string can contain an unpaired surrogate, or an upstream producer can emit invalid bytes. Naming the input UTF-8 does not make all possible byte sequences valid UTF-8.
The Web's TextDecoder offers error handling that either replaces malformed input or throws when fatal mode is selected. TextEncoder receives a scalar-value string at its API boundary; unpaired surrogates in a JavaScript input are replaced before encoding. If you see � in an output, establish whether it was already present or was introduced as replacement during an earlier operation.
Choose a policy that fits the task. A data import may benefit from stopping with a useful diagnostic so the source can be repaired. A damaged log preview may benefit from showing the recoverable portions. In either case, document the choice and retain the evidence needed to investigate it. Re-encoding a replacement character later cannot reconstruct bytes that have already been discarded.
A network chunk is not a text boundary
Suppose a reader receives two bytes at a time. The UTF-8 sequence for 中 requires three, so the first chunk contains only part of the value. That does not necessarily mean the stream is damaged: the next chunk may complete it. Creating a fresh decoder for every chunk can turn a valid stream into a series of apparent errors.
A streaming decoder retains the state needed to complete sequences across chunk boundaries. TextDecoder exposes this through its stream option, followed by a final decode call when input ends. This does not require treating the entire file as one indefinitely growing buffer. It requires maintaining decoding state while applying sensible resource limits to the overall operation.
Test this by deliberately splitting input inside a multibyte sequence and comparing the result with a single decode of the complete byte array. An all-ASCII fixture will miss the problem. Add a supplementary character and an accented sequence as well, then test later text operations independently. Correct decoding proves that the bytes became the intended sequence; it does not prove that the next truncation step preserves complete graphemes.
Transcoding, normalization, and recovery do different jobs
Transcoding changes a representation while preserving the scalar-value sequence. Normalization can change that sequence to an equivalent normalized representation. For example, NFC can turn e followed by a combining acute accent into the precomposed é. Both operations may alter byte length, but they do so for different reasons. A smaller output file does not identify which operation took place.
Recovering text decoded with the wrong encoding is another problem again. If the original bytes were interpreted incorrectly and the resulting wrong characters were then saved as valid UTF-8, the new file can be perfectly well formed and still say the wrong thing. A validator can check byte structure; it cannot infer an author's intended words from that fact alone.
A good conversion workflow therefore starts with evidence about the source. Keep a representative sample, inspect the original bytes where available, and compare converted names and symbols with a trustworthy reference. Make transformations explicit in the tool's interface. A button called “fix encoding” conceals too many possible operations; a clear preview lets users see what will actually change before replacing their data.
Choose for the boundary, then measure the actual result
UTF-8 is a sensible starting point for new general-purpose Web text interchange; the WHATWG Encoding Standard directs new formats and protocols toward it. Existing interfaces may require something else. Follow the established contract and use reliable conversion at the boundary rather than trying to force every internal runtime to represent strings in the same way.
Before shipping, answer four questions: Which encoding carries the external bytes? What unit do internal indices use? What happens to malformed input? What does the visible character limit mean? Then run representative examples through the entire path. Record bytes, code units, code points, and grapheme clusters separately instead of assuming one count predicts the others.
The byte examples here were checked in Node.js 22. You can repeat them with your own text or inspect it in the Unicode inspector. The purpose is not to crown one UTF as universally superior. It is to make each boundary explicit enough that a surprising result has a testable explanation. Once the units and error policy are clear, most encoding debates become much more concrete.