A model never sees text — it sees token IDs. The same text can be chopped into tokens many different ways, and a filter that watches IDs is blind to the string those IDs spell. You'll use that.
The word forbidden must reach the user, but the token filter must read ALLOWED. The filter blocks the exact token IDs of the canonical encoding. Find another encoding of the same text.
One of these tokens is in the vocabulary but almost never appeared in training — a glitch token. Click the one that never shows up in the sample text below.
Why it works: decoding is many-to-one — a string has many tokenizations, only one canonical. Filter the IDs and you filter one of them. Glitch tokens exist because the tokenizer's vocab and the model's training data disagree. Fix: filter the decoded, normalized string; audit the vocab for undertrained tokens. Build the real thing →