The generator that builds the card posted to social media when a blog entry goes out was drawing little boxes. Not always, but often enough that I had to inspect every image before approving it. Commit 6995d6c is called "Ask the font, not a range, whether it can draw a character", and that pretty much sums up the fix. Here's the why, which is the useful part.
The problem: the little box
When a font has no glyph for a character, the rendering engine doesn't fail: it draws .notdef, which in most fonts is an empty rectangle. People call it tofu. On a 1080-pixel card heading to Facebook and Instagram with the entry's title across it, a tofu in the middle of a word is impossible to miss.
The title is written by a model and reviewed by me, so all sorts of things come through: curly quotes, em dashes, single-character ellipses, the odd currency symbol, a stray arrow. Perfectly legitimate things in a Spanish text, but not every font covers them.
The first solution, which was the bad one
My first go was a whitelist of Unicode ranges. Printable ASCII, the Latin-1 supplement for accented vowels and the ñ, a handful of general punctuation marks. Anything outside that got stripped before painting.
It works 90 % of the time and fails in both directions at once:
- False positives. A character being in an "allowed" range doesn't mean that particular font has it. Latin-1 includes the Icelandic Þ and the pound sign; plenty of web-subset fonts don't ship the whole block. The range said yes, the font said tofu.
- False negatives. The other way round: perfectly drawable characters were dropped from the card because I hadn't added their range to the list. That's worse than the box, because it's silent: the word comes out misspelled and nobody notices.
On top of that, the list was code that had to be maintained. Every time a new character showed up I had to open the Unicode tables and add a range by hand. That alone is a sign the approach is wrong.
The solution: ask the font
A font file contains a table, the cmap, which is exactly the map from code points to glyphs. It exists for this. The question "can you draw this character?" has a direct answer: look up the code point in the cmap and, if the glyph id you get back is 0 — the .notdef — it can't.
So the whitelist went away and in its place there's a function that walks the text and asks, character by character, the very font that will do the painting. No ranges, no assumptions. The font is the only thing with the correct information, because it's the one doing the work.
Two details that took a while:
- You have to iterate code points, not UTF-16 units. In JavaScript, looping over a string by index cuts any character outside the basic plane — emoji, mostly — clean in half, and you end up asking about two halves that don't exist. With
for...oforArray.fromthe string is walked by code point and the problem disappears. - The
cmapanswers per character, not per sequence. An emoji with a variation selector, or a family joined with a zero-width joiner, is several code points that only mean something together. The font can have the pieces and still not know how to compose them.
What I do when the answer is no
Dropping the character outright throws information away. The generator's order is:
- If there's a reasonable equivalent, substitute it: curly quotes for straight ones, em dash for hyphen, single-character ellipsis for three dots.
- If there isn't, remove it — better a gap than a black rectangle.
- Either way it gets logged, and that note lands in the approval email I read before the card goes anywhere. If a title loses a meaningful symbol, I find out before publishing, not after.
What this doesn't fix
So it doesn't catch you out:
- It doesn't render emoji. Knowing the font lacks the glyph doesn't hand you the glyph. Colour emoji need a different font and an engine that supports colour glyphs; that isn't set up and right now I don't need it.
- It doesn't do shaping. In Arabic, Devanagari or any script with contextual forms, having all the glyphs isn't enough: you have to decide which shape each letter takes based on its neighbours, and that's a job for something like HarfBuzz. Checking the
cmapis nowhere near a substitute. - It doesn't measure. A character being drawable doesn't mean the title fits. Text overflow on the card was a separate fix, the commit right next to this one.
- It doesn't judge quality. A font can have a hideous glyph for some obscure symbol. The
cmaptells you it exists, not that it's presentable.
The rule I'm taking away
When you want to know whether something works, ask the thing that's going to do it, not a table describing how the world ought to be. It's the same idea as feature detection in the browser instead of sniffing the user agent, or calling the API and seeing what comes back instead of trusting the docs. Range lists, compatibility maps and "X supports this" tables age badly and lie without warning. The font, the API or the browser don't.
Any questions, tell me and we'll look at it.
Best, Vicente.