Replacement character, U+FFFD

The character a decoder substitutes when it meets bytes it cannot interpret. It renders as a question mark in a diamond, and it is the visible symptom of an encoding problem that already happened.

How it works

U+FFFD is what a decoder substitutes when it encounters a byte sequence that is not valid in the encoding it was told to assume. It is not a character that went wrong; it is a marker showing that one already did, and by the time it appears the original bytes have been discarded. The usual cause is a mismatch: a file written as UTF-8 and read as Latin-1 or the reverse, or a database column whose declared encoding differs from what was inserted. Because decoders emit it during reading rather than writing, storing text containing it converts a transient failure into a permanent one, since re-encoding writes the replacement character itself as valid UTF-8 and the loss can no longer be undone by fixing the encoding upstream.

What it is legitimately for

As a signal during decoding. It is not meant to be stored or forwarded, only to show that something went wrong.

What it breaks

• It is already the evidence of loss: the original character is gone, not hidden.
• Stored and re-encoded, it becomes permanent corruption that no later fix can undo.
• It breaks comparison with the correct text, and looks like carelessness to a reader.

How to find it yourself

You can see it. ef bf bd in a hex dump.

What Clipboard Sanitizer does

Removed. It carries no information, and leaving it in a document only propagates a failure that already occurred.