# Why OCR loses the text on buttons, and what brings it back **Disclosure:** we built and sell the Chrome extension this post mentions at the end, [Offline OCR](https://chromewebstore.google.com/detail/mfaolaapfbdmfilfgjgcpkdgfimljdpj) (publisher WrenTools). The post was written with AI assistance and reviewed by a person before publishing. Every output below was produced by one script, run on our own machine against a screenshot we drew ourselves; every name and number in it is made up. **In short:** Tesseract works in two steps. It first finds where the text is, on a black-and-white copy of your image, and only then reads each piece, from the grey. On a made-up screenshot it read 14 of 20 words: a blue button and a red badge became solid black slabs, the layout step found no text in their rows, and the plain "Cancel" and "Paid" beside them were lost too. One flag, `-c thresholding_method=2` (Sauvola, Tesseract 5), brought it to 18 of 20; shading each panel to its own background before OCR read all 20. ## The question In 2016 someone asked on tesseract.js's tracker, ["Does not get text from image if image contains some background objects"](https://github.com/naptha/tesseract.js/issues/19). A quote on a plain background came back perfectly. The same kind of quote over a picture came back as this: ```text w?- 3 <5" I! r - WRI‘Cf' v EALWAvs“ ``` They asked "if the lib only works with image without any background". Ten years later the same thing happens to the text on buttons in a screenshot, for the same reason, and the reason is worth knowing because it tells you which fixes can work. ## The test image A 420 by 256 pixel crop of a billing page we drew, with seven pieces of text (20 words): a heading, a grey sentence, a blue "Save changes" button beside a light grey "Cancel" button, a pale green "Paid" badge beside a red "3 overdue" badge, and a dark toast reading "Invoice INV-2027-0418 sent". Before OCR it is enlarged 2x and turned grey, as our own extension did before reading. Panel a of the figure further down is the crop. ## The obvious way, and what it returns ```bash tesseract in.png stdout --oem 1 --psm 3 ``` ```text Billing settings Your next invoice is due on 1 March 2027. Invoice INV-2027-0418 sent ``` Both buttons and both badges are gone. The white-on-dark toast survived, so it isn't simply light text Tesseract can't read. And "Cancel" and "Paid" are ordinary dark text on a light background, the easiest case there is. Tesseract can save the black-and-white image it works from: add `-c tessedit_write_images=1` and it writes `stdout.processed.tif` beside the output. **Predict:** in that black-and-white image, which of the four lost texts can't be read any more? Decide before you look at panel b. ![Four panels. a: the screenshot crop, a billing page with a blue Save changes button, a grey Cancel button, a green Paid badge, a red 3 overdue badge and a dark toast. b: Tesseract's own black-and-white copy, where the blue button, the red badge and the toast are solid black with white text, and all four button and badge texts were lost. c: the same with Sauvola thresholding, where the slabs become outlines and 18 of 20 words are found. d: each panel shaded to its own background first, every text dark on white, 20 of 20 words found.](https://cdn.hashnode.com/uploads/gql/6ad91735786cf66498f3c2f4/87488b17-cbaf-4b20-8949-b5e6f07e3298.png) None of them. In panel b, "Cancel" and "Paid" are crisp black on white, and "Save changes" and "3 overdue" are crisp white on black, exactly like the toast that was read. The letters weren't destroyed. Tesseract never looked for them. ## Two steps: find, then read A Tesseract contributor laid this out in [an issue about its binarization](https://github.com/tesseract-ocr/tesseract/issues/3083): since Tesseract 4, recognition "uses the greyscale-converted raw image", but "the binarized image is still needed for all segmentation and layout analysis", and that binarized image comes from a "custom global Otsu implementation": one threshold for the whole picture. So a word is read only if the layout step drew a box around it first. On our crop it drew 3 text blocks (`tesseract in.png stdout --oem 1 --psm 3 tsv`, rows of level 2): the heading, the sentence and the toast. The rows with the buttons and the badges got no block at all, so there was nothing to read. The plain words are the surprising part, so we tested them on their own. We drew the crop again without the blue button, and "Cancel" was read. Without the red badge, "Paid" was read. A filled slab takes the plain word beside it down with it. Why the slab in a row of two does this and the toast, alone in its row, doesn't, we can't tell from outside; Tesseract doesn't report it. ## Fix 1: a local threshold (one flag) Tesseract 5 added a choice of binarizer. `tesseract --print-parameters` lists it as `thresholding_method: 0 = Otsu, 1 = LeptonicaOtsu, 2 = Sauvola`. Sauvola is one of Leptonica's adaptive methods: it sets the threshold from each pixel's neighbourhood instead of from the whole image. ```bash tesseract in.png stdout --oem 1 --psm 3 -c thresholding_method=2 ``` ```text Billing settings Your next invoice is due on 1 March 2027. Save changes Cancel nM 3 overdue | oice INV-2027-0418 sent ``` 18 of 20. Panel c shows why: with a local threshold, a slab's inside is compared with itself, so it turns white and only the edges, the panel's and each letter's, stay black. The layout step now sees letter-sized shapes inside an outline. The letters in that copy look mangled, and "Save changes" was still read perfectly, because the reading step works from the grey image, not from that copy. The price: "Paid" is lost, the toast lost "Inv", and two scraps of noise ("nM", "|") came from the outlines. ## Fixes that don't work as well If layout is the problem, why not skip it? Every row below reads the same grey image: | What we tried | Words found | Texts lost | |---|---|---| | Default (`--psm 3`, Otsu) | 14/20 | Save changes, Cancel, Paid, 3 overdue | | `--psm 6`, one block, no layout analysis | 11/20 | the same four, and the toast | | `--psm 11`, sparse text | 12/20 | Save changes, Cancel, 3 overdue, the toast | | `--psm 12`, sparse text with orientation detection | 18/20 | 3 overdue (and it added "Gat \|" and ":") | | `thresholding_method=1` (Leptonica's Otsu) | 16/20 | Save changes, Cancel, Paid (and "COE", "CEs" added) | | `thresholding_method=2` (Sauvola) | 18/20 | Paid; "Inv" of the toast | | The whole crop inverted | 14/20 | the same four as the default | | Each panel shaded to its own background | 20/20 | none | Skipping layout analysis made it worse, and the sparse modes swing from 12 to 18 words on one image. Inverting the whole crop changes nothing, because the page and the slabs simply swap colours and the contrast between them stays the same. ## Fix 2: give every text the same background The last row is the one that read everything, and it's image code you can write yourself (about 160 lines of JavaScript, comments included, in our extension): 1. Take the page's colour from the crop's border (its most common colour there). 2. Group neighbouring pixels of nearly the same colour into flat regions. 3. Call a region a panel if it's large, mostly filled, far from the page's colour, and has something inside it. 4. Shade every pixel by its distance from the background it sits on: the page for most pixels, its panel's colour for pixels inside a panel. Text becomes dark, every background becomes white. Panel d is the result: a page of dark text on white, with no slabs left for the layout step to trip on. Text inside a panel was always there in the grey image; this just stops the black-and-white copy from hiding it. ## What this doesn't fix, and what we got wrong Our own extension had this bug. Up to version 0.1.1 it turned the crop grey and nothing else, and on our benchmark of rendered fixtures the "buttons" set read at a character error rate of 0.692. With the shading step, 0.000. It doesn't help layout of a different kind: two-column text and dashboards still read at 0.58 and 0.55 on the same benchmark, because there every word is found and then put in the wrong order. These numbers are for one made-up crop in one font family, read by Tesseract 5.4. Another image will move them, which is why the table shows what was lost and not just a score. ## Questions people ask **Should I invert dark screenshots before OCR?** For a whole dark page there's no need in our tests: the benchmark's ten dark fixtures read with no errors, and [a thread on white text on dark grey](https://github.com/tesseract-ocr/tesseract/issues/2052) reports the same. For a light page with dark panels, inverting the whole image only swaps which part is dark: 14/20, the same four lost. **Does this apply to tesseract.js?** It runs the same Tesseract engine compiled to WebAssembly; Offline OCR uses tesseract.js 7.0.0, and its benchmark above was measured in the browser. The tables in this post were measured with the Tesseract 5.4 command line. ## Try it [Offline OCR](https://chromewebstore.google.com/detail/mfaolaapfbdmfilfgjgcpkdgfimljdpj) is a Chrome extension that copies the text from any part of a web page: press Alt+Shift+O, drag a box, and the OCR runs on your own computer. From version 0.1.2 it shades buttons, badges and toasts to their own background before reading, the step above. One honest limit: two-column pages can still come out in the wrong order. It's free. Which one bit you: the button, the badge, or the plain word beside it? Tell us in the comments.