In short: feedparser is the right tool for RSS and Atom, but four things catch people. It returns zero entries for a JSON Feed, so read that format with json yourself. A str you pass it is fetched as a URL or opened as a file, so pass bytes. entry.get("updated_parsed") quietly returns the published date when the feed gave no updated date; use dict.get(entry, "updated_parsed"). An Atom entry without its own author doesn't inherit the feed's. To find a site's feed in the first place, read the <link rel="alternate"> tags on its home page.

Reading a feed in Python usually starts and ends with feedparser.parse(...). It handles RSS 0.9x to 2.0, RDF and Atom, tolerates the broken XML you meet in the wild, resolves relative links and strips scripts from HTML. We built a feed reader on it. Here are the four surprises it had to handle, what feedparser actually returned for each, and the fix, then how to find a feed and filter it by date.

Predict: in our three test feeds (one RSS, one Atom, one JSON Feed), no item names its own author, and only the Atom entry gives an updated date. Once our reader has turned each first item into the same row, which rows have an author? Guess, then check the figure.

Why does feedparser return no entries for a JSON Feed?

Because it doesn't read the format. JSON Feed (1.0 and 1.1) is a third feed format, alongside RSS and Atom. Given a valid JSON Feed 1.1 document with one item:

feedparser.parse: version='' entries=0 bozo=1

No error is raised, so a loop over entries just does nothing. It has been asked for on feedparser's tracker: Support for JSON Feeds. The fix is short, because JSON Feed is plain JSON. Check for the version URL, then map the fields:

doc = json.loads(body.decode("utf-8-sig"))
if "jsonfeed.org/version/" not in str(doc.get("version", "")) or not isinstance(doc.get("items"), list):
    raise ValueError("JSON, but not a JSON Feed")
feed_authors = doc.get("authors") or ([doc["author"]] if isinstance(doc.get("author"), dict) else [])
for item in doc["items"]:
    authors = item.get("authors") or ([item["author"]] if isinstance(item.get("author"), dict) else []) or feed_authors
    link = urljoin(base_url, item["url"]) if isinstance(item.get("url"), str) else None
    published = _iso_text(item.get("date_published"))  # to UTC
    ...

Two details matter. Version 1.0 used a single author object and 1.1 uses an authors list, so read both. And 1.1 says an item with no authors takes the feed's. Our reader returned this for the same document:

format=jsonfeed entries=1 link=https://m.example/p/1 published=2026-10-21T14:30:00Z author=Sam Ito summary='First post'

The relative /p/1 became a full link, 09:30-05:00 became UTC, and the author came from the feed's top level.

Why does feedparser.parse make a network request?

Because a str argument is read as a URL, a file path, or a document, in that order. We started a server on 127.0.0.1 and passed its address as a string, then wrote a feed to a temporary file and passed its path:

feedparser.parse(url string): server requests=1 entries=2
feedparser.parse(path string): version='atom10' entries=1

That is handy in a notebook. In a service it means a string from a user can make your process fetch any URL, or read a local file, without going through your HTTP client and whatever limits you put on it: timeouts, size caps, robots.txt, a block on private addresses. Our reader fetches with its own client and passes feedparser the bytes only. The parser refuses anything else, so a refactor can't bring the old path back:

def parse(body: bytes, content_type: str, base_url: str, include_content: bool = False) -> Feed:
    if not isinstance(body, bytes):
        raise TypeError("parse takes the fetched bytes, never a URL or a path")
    ...
    parsed = feedparser.parse(body, response_headers={"content-location": base_url,
                                                      "content-type": content_type or "application/xml"})

Passing content-location keeps relative links working: the /posts/spring link in our test RSS came back as https://blog.example/posts/spring.

Why does an RSS item have an "updated" date the feed never gave?

Because feedparser maps a missing updated_parsed to published_parsed. Its own source says so, in a DeprecationWarning: "a temporary mapping has been created from updated_parsed to published_parsed if updated_parsed doesn't exist. This fallback will be removed in a future version of feedparser." For an RSS item with only a <pubDate>:

e.get('updated_parsed') is set: True; 'updated' in e: False; dict.get: None

So in says there is no updated date and .get() returns one anyway. Any code that flags items as "edited since publishing" will flag every one. Read the key without the mapping:

"published": _iso(dict.get(e, "published_parsed")),
"updated": _iso(dict.get(e, "updated_parsed")),

That gave published=2026-10-14T09:00:00Z updated=None, with the +0200 in the pubDate converted to UTC. Because the fallback is marked for removal, code that relies on it will also change behaviour on some future upgrade.

Where did the Atom author go?

An Atom feed often names its author once, on the <feed>, and not on each <entry>. RFC 4287 (s.4.2.1) says such an entry takes the feed's author. feedparser leaves the entry empty:

feedparser entry author: None; feed author: 'Dana Reyes'

Apply the rule yourself: the entry's author, then its <source>'s, then the feed's.

feed_author = parsed.feed.get("author") if parsed.version.startswith("atom") else None
...
"author": e.get("author") or (dict.get(e, "source") or {}).get("author") or feed_author,

Our reader then returned 'Dana Reyes' for the entry. We don't do the same for RSS: RSS 2.0 has no feed-level author. Its managingEditor is an editorial contact, and copying it onto every item would credit the wrong person.

How do you find a site's feed from its home page?

Many sites announce it in the page's <head>:

<link rel="alternate" type="application/rss+xml" title="Posts" href="/feed.xml">
<link rel="alternate" type="application/atom+xml" href="https://blog.example/atom.xml">
<link rel="alternate" hreflang="de" href="/de/">

The third tag is alternate too, but a translation, not a feed, so check the type as well as the rel. The standard library's HTMLParser is enough. Collect the <link> tags whose rel includes alternate and whose type is RSS, Atom, RDF or JSON Feed, make each href absolute, and drop duplicates:

discover: ['https://blog.example/feed.xml', 'https://blog.example/atom.xml']

Our reader does this when a URL you give it turns out to be HTML, and reads up to five feeds a page advertises.

A date cut-off that doesn't lose posts

For a daily digest you want "items from the last day" only. Two choices decide whether that works. Compare in UTC, after converting each item's own offset. And decide what to do with an item that has no date: a cut-off can't judge it, so we keep it rather than drop it silently. On our test feed (one item from 14 October, one undated), measured at noon UTC on 25 October:

'7 days' -> cut-off 2026-10-18T12:00:00+00:00; kept ['Undated note']
'2026-10-14' -> kept ['Spring survey', 'Undated note']

An item uses its published date, else its updated date. If you'd rather drop undated items, filter them after the cut-off, where it's visible.

Questions people ask

Does feedparser sanitize HTML? Yes. Given <p onclick="x()">All six sites.</p><script>alert(1)</script> as an item's description, its summary was <p>All six sites.</p>: the script and the event handler were gone. We also turn the summary into plain text (at most 2,000 characters) so it can go in a spreadsheet, which gave All six sites.

Will these still hold after an upgrade? We measured 6.0.14, the version we pin. The updated_parsed mapping above is marked for removal, so re-check that one when you upgrade.

Try it

The reader is about 240 lines of core.py on top of feedparser, and returns one row per item in the same shape for every format, with the feed's title on each row so many feeds can be mixed. Failures (a page with no feed, a robots.txt that disallows the feed) come back as error rows and aren't charged. It costs $0.001 a feed plus $0.0003 an item. Two examples use it: RSS feeds into a spreadsheet, which on each later run adds only the new posts, and a podcast's episodes into searchable transcripts, which reads each episode's audio link from the feed.

Which feed broke your parser: a JSON Feed, a missing date, or something stranger? Tell us in the comments.

In short: stream each download and check four things before you keep it. Refuse a file whose declared Content-Length is over your cap, and count bytes as you read in case there is no length. In our test, a 50 MB file with no length stopped after 5,242,880 bytes against a 5 MB cap. Check the Content-Type before reading the body: a "PDF" link that answers with a sign-in page is caught there. Take the file name's last path segment only, and check the address you actually connected to, because localhost passes a URL check and still reaches 127.0.0.1.

A list of links looks harmless: loop over it and call urlopen. Then one link turns out to be a 4 GB archive. Another "PDF" is an HTML sign-in page, because the share link expired. A server suggests the name ../../etc/passwd. And one URL points inside your own network, at a router or a cloud metadata address. If the list comes from someone else, that last one is a server-side request forgery (SSRF).

Here is how our downloader handles each case, using only the standard library, and what each check did when we pointed it at a test server.

Predict: six test links, one trap each: a 50 MB file with a Content-Length, the same without one, a "PDF" that answers with a sign-in page, a server-suggested name of ../../etc/passwd, a localhost address, and a redirect to ftp://. A plain loop calls urlopen, reads each body and saves it under the server's name. How many of the six does it stop? Guess, then check the figure.

How do you stop a huge file before it fills your disk?

Two checks, because a server doesn't have to tell you the size. First, read the declared length and refuse before touching the body. Then count while you read, in chunks, and stop the moment the count passes the cap:

CHUNK = 256 * 1024

declared = headers.get("content-length", "")
if declared.isdigit() and int(declared) > max_bytes:
    got.error, got.message = "file-too-large", f"{int(declared)} bytes declared"
    return got
parts, size = [], 0
while True:
    chunk = resp.read(CHUNK)
    if not chunk:
        break
    size += len(chunk)
    if size > max_bytes:
        got.error, got.message = "file-too-large", f"more than {max_bytes} bytes"
        return got
    parts.append(chunk)

Our test server streams 64 KiB blocks until it passes 50,000,000 bytes, 50,003,968 in all. We fetched that file twice with a 5,000,000-byte cap, once with Content-Length: 50000000 declared and once with no length:

The second figure is 20 chunks of 256 KiB, the first count past 5,000,000. So the most you ever hold is the cap plus one chunk. Send Accept-Encoding: identity too. urllib doesn't decompress for you, so this asks for the file itself rather than a gzip stream, and the bytes you count are the file's bytes.

How do you know you got a PDF and not a sign-in page?

Ask for the content type before you read the body. Rules are full MIME types (application/pdf) or prefixes ending in a slash (image/):

def type_allowed(content_type: str, allowed: list) -> bool:
    if not allowed:
        return True
    content_type = (content_type or "").lower()
    for rule in allowed:
        rule = rule.strip().lower()
        if rule and (content_type == rule or (rule.endswith("/") and content_type.startswith(rule))):
            return True
    return False

Our test server answered /report.pdf the way an expired share link does: status 200, text/html, "Please sign in". With ["application/pdf"] the download stopped with content-type-not-allowed, "content type text/html", and an empty body. Without the filter, those 40 bytes of HTML were stored as report.pdf. Status 200 only means the server answered. It doesn't mean you got the file.

Keep a SHA-256 of every body you store (hashlib.sha256(body).hexdigest()). It lets you skip unchanged files on the next run and spot two links that serve the same thing.

Which file name should you save it under?

Prefer the server's Content-Disposition name, else the URL's last path segment. Then keep only the last segment of whatever you got, whichever way its slashes lean. The standard library's email.message.Message already decodes the RFC 2231 filename*= form, so there's no need to parse it yourself:

def file_name(final_url: str, headers: dict, content_type: str) -> str:
    name = None
    disposition = headers.get("content-disposition")
    if disposition:
        msg = Message()
        msg["content-disposition"] = disposition
        name = msg.get_filename()
    if not name:
        name = unquote(urlsplit(final_url).path.rsplit("/", 1)[-1])
    name = (name or "").replace("\\", "/").rsplit("/", 1)[-1].strip()
    if not name:
        name = "file" + (mimetypes.guess_extension(content_type or "") or "")
    return name

What it made of the headers we sent:

The stored key adds the link's position, so two files both called report.pdf don't overwrite each other. It also replaces every character the storage doesn't allow (Apify's key-value store takes letters, digits and !-_.'()). If you write to disk instead, the position prefix still does the job.

Why check the address after connecting, not the URL?

Checking the URL's host against a blocklist misses names. localhost, or any domain whose DNS points at a private address, sails through a string check. A redirect can also change the host after your check has run. So the downloader checks the peer address of the socket it actually opened, on every connection, redirects included:

def is_public_address(ip: str) -> bool:
    addr = ipaddress.ip_address(ip.split("%", 1)[0])
    if addr.version == 6 and addr.ipv4_mapped:
        addr = addr.ipv4_mapped
    return addr.is_global


def _connect_public(address, *args, **kwargs):
    sock = socket.create_connection(address, *args, **kwargs)
    peer = sock.getpeername()[0]
    if not is_public_address(peer):
        sock.close()
        raise BlockedAddress(f"blocked-non-public-address {peer}")
    return sock

http.client sets _create_connection on each connection object, so a small subclass swaps in _connect_public, and an HTTPHandler/HTTPSHandler pair hands those subclasses to urllib. Both test URLs were refused, with nothing sent beyond the TCP handshake:

• http://127.0.0.1:<port>/report.pdf: blocked-non-public-address 127.0.0.1
• http://localhost:<port>/report.pdf: blocked-non-public-address 127.0.0.1

And what ipaddress.is_global says, after unwrapping IPv4-mapped IPv6:

Without the unwrapping step, ::ffff:127.0.0.1 is the classic way past a check that only knows IPv4 ranges.

Don't follow redirects to other schemes

urllib's redirect handler follows ftp:// as well as http(s)://. We pointed a link at a 302 to ftp://127.0.0.1/secret.txt. Plain urllib followed it and tried to open an FTP connection, which failed only because no FTP server was listening. The downloader's handler refuses anything but http and https, so the link fails as a fetch error instead:

class _HttpOnlyRedirects(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        if not newurl.lower().startswith(("http://", "https://")):
            raise urllib.error.HTTPError(newurl, code, "redirect to a non-http(s) URL refused", headers, fp)
        return super().redirect_request(req, fp, code, msg, headers, newurl)

Build the opener with ProxyHandler({}) too, so an environment variable can't quietly route your requests through a proxy.

Be polite to the sites

Fetching a list is not crawling, but the files still live on someone's server. Before the first file from a site, the downloader reads that site's robots.txt and skips what it disallows. A 401 or 403 on robots.txt, or a server error, is read as "disallow everything", the cautious reading. It also leaves 500 ms between two requests to the same site, and fetches each duplicate URL once.

What this doesn't do

• It doesn't crawl. Only the URLs you list are fetched, one request each, and links inside files are not followed.
• It doesn't sign in. A file behind a login is out of reach, by design.
• It doesn't run JavaScript. A "download" button that builds its link in the browser has no direct URL to give it.

Try it

The checks above are about 150 lines of core.py plus 165 of http_fetch.py, standard library only. The Actor stores each file in the run's key-value store and returns a dataset row per link with the file name, size, content type, SHA-256, or the reason it failed. Failed links aren't charged. It costs $0.002 a file plus $0.001 for each started megabyte. The GitHub example reads a text file of links, downloads them in one run, checks each SHA-256 and keeps a manifest, so a second run fetches only new or fixed links.

Which of these has bitten you in a real script: the huge file, the sign-in page, the odd file name, or the private address? Tell us in the comments.

In short: a PDF has no headings or paragraphs, only positioned text with font sizes. To get Markdown, take pdfminer.six's text boxes, call the most common size (weighted by characters) the body, and make short blocks at least 1.12 times larger into #, ## and ###, largest first. Then join wrapped lines, turn bullet lines into - items, and escape lines that only look like Markdown. On our three-page test report that gives five headings, a list and two clean paragraphs.

A PDF is a page description, not a document. It says "draw these glyphs at this position in this font at this size", and nothing marks a line as a heading or says where a paragraph ends. If you want Markdown, for a notes app, a static site or a RAG pipeline, you have to infer the structure back from the layout.

Here is what that looks like in Python with pdfminer.six, using the rules from our extractor's core.py.

What does plain text extraction give you?

pdfminer's one-liner on our test report:

from pdfminer.high_level import extract_text
extract_text("report.pdf")

'Quarterly Field Report\n\n1. Summary\nThe survey covered twelve sites over six weeks. Most sites met the target, and two were\nre-measured after the storm in week four.\n\n2. Findings\n\n2.1 Water quality\n...'

The words are all there, but a line break falls in the middle of a sentence, page breaks show up as \x0c form feeds, and nothing tells "2.1 Water quality" apart from the body text. The structure is in the font sizes, so we need the layout objects instead.

1. Read text boxes with their sizes

pdfminer's layout analysis groups characters into LTTextBox objects made of LTTextLines, and each character (LTChar) knows its size and font name. Per line we keep the median size and whether most characters are bold:

from statistics import median
from pdfminer.layout import LTChar

def line_style(line):
    chars = [c for c in line if isinstance(c, LTChar) and c.get_text().strip()]
    if not chars:
        return 0.0, False
    size = median(round(c.size, 1) for c in chars)
    bold = sum("bold" in c.fontname.lower() for c in chars) > len(chars) / 2
    return size, bold

The median keeps one odd glyph, such as a superscript, from moving the line's size.

2. Split a box when its lines change size

pdfminer often puts a heading in the same box as the paragraph right under it. So we walk the lines of each box and start a new block whenever the size changes by 0.6 pt or more, or the weight changes:

current = None
for text, size, bold in styled_lines:
    if current and abs(current.size - size) < 0.6 and current.bold == bold:
        current.lines.append(text)
    else:
        current = Block(page, [text], size, bold)
        blocks.append(current)

3. Find the body size

The body text is whatever most of the characters are set in. Count characters per (rounded) size, not lines or blocks, because one long paragraph outweighs ten short captions:

from collections import Counter

def body_size(blocks):
    weights = Counter()
    for b in blocks:
        weights[round(b.size)] += sum(len(line) for line in b.lines)
    return float(weights.most_common(1)[0][0]) if weights else 0.0

Predict: the test report sets its text at four sizes: 22, 15, 12.5 and 11 pt. Which one will body_size pick as the body, and which of the others will end up as #, ## and ###? Make a guess, then check it against the figure.

On the test report that gives 11 pt.

4. Map larger sizes to heading levels

Every size at least 1.12 times the body is a heading size. The largest becomes #, the next ##, and everything smaller than that ###:

HEADING_RATIO = 1.12

def heading_levels(blocks):
    body = body_size(blocks)
    sizes = sorted({round(b.size) for b in blocks if body and b.size >= body * HEADING_RATIO}, reverse=True)
    return {size: min(i + 1, 3) for i, size in enumerate(sizes)}

The report has headings at 22, 15 and 12.5 pt, so the result is {22: 1, 15: 2, 12: 3}. The 12.5 pt subheadings are only 1.14 times the body, so a threshold much above 1.12 would have missed them. A block also has to be short to count, at most 3 lines and 200 characters, so a paragraph set slightly large (a pull quote, an abstract) stays a paragraph, but only while pdfminer keeps its lines in one box. In our test, a 4-line 14 pt paragraph with lines 1.2 times its size apart stayed one paragraph; spaced twice its size apart, each line became a box of its own and a ## heading.

5. Join wrapped lines, and rejoin hyphenated words

Inside a paragraph block, lines are joined with a space. A line that ends in - followed by one that starts with a lowercase letter is treated as a word broken across the line:

def join(lines):
    out = ""
    for line in lines:
        if not out:
            out = line
        elif out.endswith("-") and line[:1].islower():
            out = out[:-1] + line
        else:
            out += " " + line
    return out

["the extraction was effi-", "cient"] becomes "the extraction was efficient". The rule has limits. A real hyphenated word that happens to break at its hyphen loses it: ["two sites were re-", "measured"] becomes "two sites were remeasured". And ["the Covid-", "19 wave"] becomes "the Covid- 19 wave", because the digit isn't lowercase. Fixing those needs a dictionary. We decided a rare wrong join was better than leaving every broken word split.

6. Turn bullet lines into list items

A line that starts with a bullet character and a space becomes a - item. A following line that doesn't start with a bullet is a wrapped continuation of that item:

import re
BULLET = re.compile(r"^\s*[•●▪◦‣⁃–\-\*]\s+")

7. Escape what only looks like Markdown

A body paragraph that starts with 1. or # would turn into a list or a heading once rendered. So the start of each paragraph is escaped:

def escape_md(text):
    return re.sub(r"^(#{1,6}\s|>|\d+\.\s)", r"\\\1", text)

"1. Summary of costs" becomes \1. Summary of costs, and "# of sites" becomes \# of sites.

The result

The same test report, through all seven rules:

# Quarterly Field Report

## 1. Summary

The survey covered twelve sites over six weeks. Most sites met the target, and two were re-measured after the storm in week four.

## 2. Findings

### 2.1 Water quality

Turbidity fell at nine of twelve sites.

- Site A: within limits
- Site B: within limits
- Site C: above limits

### 2.2 Equipment

Two probes were replaced. The spare kit is now stored at the north depot so field crews can reach it within an hour.

"2.2 Equipment" sits on page 2, and it is still a ###, because the levels are computed over the whole document rather than page by page.

What this approach doesn't do

• Tables and images are not rebuilt. A table comes out as its cell text in reading order.
• Scanned pages have no text layer, so there is nothing to read. Page 3 of the test report is blank, and the extractor lists it as a page without text instead of returning an empty string as if it had succeeded. For scans you need OCR (we use Tesseract in a separate OCR Actor).
• Protected PDFs. pdfminer will extract text from a PDF whose owner has disabled copying unless you check first. We check PDFDocument.is_extractable and refuse those files, and we refuse password-protected ones too.

Try it

The full code is about 280 lines of core.py. It also splits the Markdown into overlapping chunks for RAG, each tagged with its start and end page. If you'd rather call it than run it, the GitHub example sends a list of PDF links to the Actor and saves Markdown files and chunks. It costs $0.002 a PDF plus $0.0001 a page.

If you've found a better signal than font size for headings, such as weight, spacing or the PDF's own tags, I'd like to hear it in the comments.

In short: a scanned PDF page has no text layer, so render it to an image with pypdfium2 and read it with Tesseract. Render at 300 dpi. On our test page, 72 dpi lost all the small print and turned "9:00" into "2:00". 150 dpi read "Fine print" as "Fine pint". 300 dpi read every line exactly, in about twice the time of 150. Ask Tesseract for TSV output: each word's confidence lets you flag the misreads it does make.

When a PDF comes from a scanner or a phone app, each page is a picture. Text extraction returns nothing, because there is no text in the file, only pixels. You have to draw each page and read the pixels back with OCR.

Here is how our OCR Actor does it in Python, and what we measured when we changed the one setting that matters most, the resolution.

How do you tell a scanned page from a text page?

Ask the PDF for its text layer. With pypdfium2:

import pypdfium2 as pdfium

pdf = pdfium.PdfDocument("scan.pdf")
page = pdf[0]
page.get_textpage().get_text_range()

''

Our test page is a Letter page scanned at 200 dpi. It holds a notice at body size, two lines of small print (about 7 pt) and one line of fine print (about 5 pt), on off-white paper with speckle and a 1.2 degree tilt, like a phone scan. Its text layer is empty, so OCR is the only way to read it.

Draw the page, then read it with Tesseract

PDF pages are measured in points, 72 to the inch, so a scale of dpi / 72 draws the page at the resolution you want. Greyscale is enough for OCR:

dpi = 300
img = page.render(scale=dpi / 72, grayscale=True).to_pil()
img.save("page.png")

Then run the tesseract program on the image. We ask for TSV rather than plain text, because TSV gives every word with its position and a confidence from 0 to 100:

import subprocess

args = ["tesseract", "page.png", "stdout", "-l", "eng", "--psm", "3", "--dpi", str(dpi), "tsv"]
tsv = subprocess.run(args, capture_output=True, timeout=180).stdout.decode("utf-8", "replace")

--psm 3 is Tesseract's automatic page layout. We pass --dpi because we know it; without it Tesseract estimates the resolution. On this page, leaving it out gave the same text, with two paragraph breaks in different places.

Which resolution? We measured it

Predict: before you look at the chart, guess two things. What happens to the 5 pt fine print at 72 dpi, and how much longer does a page take at 300 dpi than at 150? Then check your guess against the figure.

The same page, drawn at three resolutions and read with Tesseract 5.4 (English):

(Accuracy is character similarity to the text we drew. Times are on one machine and are only for comparing the rows.)

At 72 dpi, both lines of small print are simply missing, and the body text has a silent error:

‘on Tuesday 4 November from 2:00 to 13:00.
lease store drinking water in advance.

The page says 9:00. A wrong number that still looks like a number is the worst kind of OCR error, because nothing downstream notices it.

At 150 dpi the body and the small print are right, and the fine print comes out as:

Fine pint this offer is valid for orders placed before 31 March 2027 only

At 300 dpi every line is exact:

Fine print: this offer is valid for orders placed before 31 March 2027 only.

300 dpi took about twice as long as 150 here. We use 300 for every page, because the small print on a page is often the part someone needs, such as terms, reference numbers and dates.

Use word confidences to catch misreads

Each TSV row is one word (level 5), with its block, paragraph and line numbers, its box and its confidence. Not every misread scored low: at 72 dpi ‘on (a stray mark before "on") scored 91.4, and at 150 dpi only ("only." without its full stop) scored 82.2. The wrong time and the misread word in the fine print did:

So a confidence threshold makes a good review flag. It is not a verdict, though. At 300 dpi, "offer is" was read correctly with a confidence of 10.5. Flag low words for a person to check, and don't delete them.

We also report a character-weighted mean confidence per page (81.2 at 72 dpi, 91.4 at 150, 93.9 at 300). A low page mean is a quick sign that a page needs a second look.

Rebuild the lines from the TSV

The TSV keys (block, paragraph, line) give you the layout back. Words with the same key share a line, and a new block or paragraph starts a new paragraph:

def parse_tsv(tsv):
    words, lines, last = [], [], None
    for row in tsv.splitlines()[1:]:
        cols = row.split("\t")
        if len(cols) < 12 or cols[0] != "5":       # level 5 = a word
            continue
        text, conf = cols[11].strip(), float(cols[10])
        if not text or conf < 0:
            continue
        key = (int(cols[2]), int(cols[3]), int(cols[4]))  # block, paragraph, line
        words.append((text, conf))
        if last is not None and key[:2] != last[:2]:
            lines.append("")                         # blank line between paragraphs
        if key != last:
            lines.append(text)
        else:
            lines[-1] += " " + text
        last = key
    return words, "\n".join(lines).strip()

Guard rails worth having

• Very large pages. A poster drawn at 300 dpi can run to hundreds of megapixels. We scale anything over 40 million pixels down before OCR, and mark that page as scaled. Pillow's decompression-bomb check refuses an image file over 400 million pixels without decoding it.
• Access controls. A PDF whose owner has turned off copying (permission bit 5) is refused. Drawing it and reading the pixels back would extract the same text by a different route. A PDF that needs a password is refused too, and the tool takes no password input.
• One thread. On a machine with a fraction of a core, Tesseract's OpenMP threads only compete with each other, so we set OMP_THREAD_LIMIT=1.

What this doesn't do

• Layout. You get lines and paragraphs, not tables or columns as structure.
• Handwriting. We haven't tested it, and Tesseract is built for printed text.
• Text PDFs. If a page already has a text layer, extract it instead (see our post on rebuilding Markdown from font sizes). It's faster and it's exact.

Try it

The whole thing is about 230 lines of core.py. The Actor has 27 Tesseract languages installed and reads images (PNG, JPEG, TIFF, WebP and more) as well as PDFs. The GitHub example sends a list of PDF and image links to it and saves the text. It costs $0.003 a page.

What resolution do you scan or render at, and have you found a case where 300 dpi wasn't enough? I'd like to hear it in the comments.

In short: tesseract.js runs fully offline in a Manifest V3 extension once it lives in an offscreen document (a service worker has no DOM) and every worker, WASM and language path points inside the package, about 15 MB in all. Map the drag onto the screenshot with the image's own scale rather than devicePixelRatio, enlarge the crop to 2 pixels per CSS pixel, drop lines under confidence 60, and prove "offline" with a test that cuts the network.

We wanted a Chrome extension that reads text out of anything on screen (a paused video, a picture, a screenshot of an error) without sending the image anywhere. tesseract.js does the reading. Getting it to run fully inside a Manifest V3 extension, and read well, took five decisions we could only make by measuring. It is our answer to tesseract.js's Will tesseract.js support a Manifest V3 extension?.

1. The engine can't run in the service worker

tesseract.js runs in a Web Worker and wants a canvas. An MV3 background script is a service worker, and Chrome's docs say it plainly: "Service workers don't have DOM access." The way out is an offscreen document, a hidden page the extension owns. Only one can be open at a time, so creation has to be guarded against two clicks racing:

let creating = null;

async function ensureOffscreen() {
  if (await chrome.offscreen.hasDocument()) return;
  if (!creating) {
    creating = chrome.offscreen.createDocument({
      url: 'offscreen.html',
      reasons: ['WORKERS'],
      justification: 'Runs the bundled OCR engine in a Web Worker to read text from the selected area.',
    }).finally(() => { creating = null; });
  }
  await creating;
}

The service worker captures the tab (chrome.tabs.captureVisibleTab) and passes the screenshot and the dragged rectangle to the offscreen document, which crops, reads and replies.

2. No remote code, so the engine's defaults have to go

MV3 extensions "need to bundle all code they are using inside the extension itself", WebAssembly included (Chrome's docs). tesseract.js fetches its worker, its WASM core and its language data from a CDN by default. So every path points inside the extension (if you came from tesseract.js's Tesseract couldn't load any languages! or tesseract.js with a custom language and no internet, langPath is the line that points the engine at language data in the package):

Tesseract.createWorker('eng', Tesseract.OEM.LSTM_ONLY, {
  workerPath: chrome.runtime.getURL('vendor/worker.min.js'),
  corePath: chrome.runtime.getURL('vendor/core'),
  langPath: chrome.runtime.getURL('vendor/lang'),
  workerBlobURL: false,
  cacheMethod: 'none',
  gzip: true,
});

Setting options wasn't enough for us. The shipped scripts still contain the CDN addresses as defaults (we counted one in tesseract.min.js and two in worker.min.js). Our build replaces each with a dead local path, stops if the count ever changes, and a test fails if any built script names a remote address. The package comes to about 15 MB, nearly all of it the engine (three WASM cores, picked by the CPU's SIMD support) and the English model.

3. Mapping a drag onto the screenshot: take the scale from the image

The user drags in CSS pixels. captureVisibleTab returns device pixels. It's tempting to multiply by devicePixelRatio, but page zoom changes the ratio as well. The screenshot already knows the answer: its width over the viewport's width.

Predict: a 2x screen captures a 1000 px wide viewport at 2000 px, and you drag a 100 x 50 box at (10, 20). Where is the crop in the screenshot? And on a 1x screen, which pixel columns does a drag from x = 10.5 to x = 30.7 keep? Guess, then check the figure below the code.

function cropBox(rect, viewportWidth, imageWidth, imageHeight) {
  const s = imageWidth / viewportWidth;
  const x0 = Math.max(0, Math.floor(rect.x * s));
  const y0 = Math.max(0, Math.floor(rect.y * s));
  const x1 = Math.min(imageWidth, Math.ceil((rect.x + rect.width) * s));
  const y1 = Math.min(imageHeight, Math.ceil((rect.y + rect.height) * s));
  if (x1 - x0 < 2 || y1 - y0 < 2) return null;
  return { x: x0, y: y0, width: x1 - x0, height: y1 - y0 };
}

Flooring the start and ceiling the end means a box never clips the edge of a glyph.

4. Enlarge to 2 pixels per CSS pixel; don't bother inverting

Our first build read the crop as it came, because the first three test pages read well. Then we rendered the store screenshots and code text came out wrong. Measured on the same blocks (similarity to the true text):

2x helped or tied on every block we tried, so the crop is drawn onto a canvas in grey at 2 image pixels per CSS pixel (a 2x screen's capture is already there), capped at a 4096 px side so a large photo isn't blown up. We also tried inverting light-on-dark text first. It changed no read, because Tesseract already retries inverted lines itself, so we dropped it.

What still misreads, for honesty: at 14 px a caret can come out as a tilde (^18.0.0 as ~18.0.0), and once 18.2.0 came out as l8.2.0.

5. Drop the low-confidence lines

Cut a screenshot tight around a dark, rounded box and the corners read as junk lines (Ee ™, o 4). Their line confidence was 21 to 44. Every real line we measured scored 74 or more. So lines under 60 are left out:

const MIN_LINE_CONFIDENCE = 60;
function textOf(blocks) {
  const paragraphs = [];
  for (const block of blocks || []) {
    for (const paragraph of block.paragraphs || []) {
      const lines = (paragraph.lines || []).filter((l) => l.confidence >= MIN_LINE_CONFIDENCE)
        .map((l) => String(l.text).replace(/\n+$/, ''));
      if (lines.length) paragraphs.push(lines.join('\n'));
    }
  }
  return paragraphs.join('\n\n');
}

Bonus: proving "offline" with a test, not a promise

"Nothing leaves your computer" is easy to write. We wanted a test that fails if it isn't true. The browser tests start Playwright's Chromium with every host except the local test server resolving to nothing:

args=["--host-resolver-rules=MAP * ~NOTFOUND, EXCLUDE 127.0.0.1"]

If the engine tried to fetch anything from a CDN, the read would fail. It reads exactly. (Two practical notes: branded Chrome no longer loads unpacked extensions from the command line, so the tests use Playwright's own Chromium. And a test can't click the toolbar button that grants activeTab, so the test build alone adds host access. A static test checks that this, an open shadow root and the name are the only differences from the shipped build.)

The permissions came out small as a side effect: activeTab, scripting and offscreen, no host access and no content script. The extension sees a page only when you call it.

If you've shipped tesseract.js in an extension and found a better answer to any of these, especially the small code text, I'd like to hear it in the comments.

In short: send a list of media links to the Audio & Video Transcription Actor with the apify-client package, and each file comes back as a plain-text transcript and an .srt subtitle file, transcribed by Whisper inside the run. It costs $0.01 per started minute of audio (a one-hour meeting is $0.60), and max_total_charge_usd caps what one run can spend.

You have a folder of interviews, lectures or voice notes online and want two things for each: a plain-text transcript and an .srt subtitle file. This example does that in one Apify run, from Python, with a hard cap on what the run can cost.

What runs where

The Audio & Video Transcription Actor runs OpenAI's Whisper model (base, through faster-whisper) inside the run, so the files are not sent on to a third-party speech API. It accepts most audio and video formats (MP3, M4A, WAV, FLAC, OGG, MP4, MOV, WebM, MKV and more), detects the language, and returns the text, timed segments and subtitle files. It skips silence and music before transcribing, which reduces made-up text in long pauses.

It costs $0.01 per started minute of audio, at least one minute per file, with nothing charged for files that fail. A 30-second voice note costs $0.01 and a one-hour meeting $0.60. You need an Apify account and its API token (Console > Settings > API & Integrations).

The code

pip install "apify-client>=3,<4"
export APIFY_TOKEN=<your token>          # PowerShell: $env:APIFY_TOKEN = "<your token>"
python transcribe.py https://example.com/interview.mp3 https://example.com/lecture.mp4

import os
import sys
import urllib.request
from decimal import Decimal
from pathlib import Path

from apify_client import ApifyClient

ACTOR = "wrentools/audio-video-transcription"


def main(urls: list[str]) -> int:
    client = ApifyClient(os.environ["APIFY_TOKEN"])
    run = client.actor(ACTOR).call(
        run_input={
            "fileUrls": urls,
            "language": "auto",          # or "en", "de", "es", ...
            "subtitleFormats": ["srt"],
        },
        max_total_charge_usd=Decimal("1.00"),  # the run stops before spending more than this
    )
    if run is None or run.status != "SUCCEEDED":
        print(f"run did not succeed: {run.status if run else 'no run'}", file=sys.stderr)
        return 1

    out = Path.cwd()
    for item in client.dataset(run.default_dataset_id).iterate_items():
        if item["status"] != "ok":
            print(f"FAILED {item['source']}: {item['error']['code']}")
            continue
        stem = Path(item["fileName"]).stem
        (out / f"{stem}.txt").write_text(item["text"], encoding="utf-8")
        srt_url = (item.get("subtitles") or {}).get("srt")
        if srt_url:
            with urllib.request.urlopen(srt_url) as response:
                (out / f"{stem}.srt").write_bytes(response.read())
        print(f"{item['fileName']}: {item['language']}, {item['durationSeconds']:.0f} s, "
              f"{item['wordCount']} words, billed {item['billedMinutes']} min")
    return 0


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit("usage: python transcribe.py <media file link> [more links...]")
    sys.exit(main(sys.argv[1:]))

call() starts the run and waits for it. The run's dataset then holds one item per file, in the order you gave them. max_total_charge_usd caps the run: the Actor stops before a file it cannot pay for and transcribes only the minutes the remainder covers.

What comes back

For a 24-second sample recording, the item looks like this (shortened):

{
  "fileName": "speech-sample.mp3",
  "status": "ok",
  "language": "en",
  "durationSeconds": 24.005,
  "billedMinutes": 1,
  "wordCount": 59,
  "text": "This is a short sample recording for the transcription actor. The Riverside Community Garden will hold ...",
  "segments": [
    {"id": 0, "start": 0, "end": 4, "text": "This is a short sample recording for the transcription actor."}
  ]
}

and the .srt the script writes starts:

1
00:00:00,000 --> 00:00:04,000
This is a short sample recording for the transcription actor.

A file that fails has "status": "error" and an error.code such as fetch-failed, no-audio-track or unsupported-codec.

Options worth knowing

• language: auto, or a code such as en or ja. Setting it skips detection.
• task: "translate": English text from speech in any language (rougher than transcription).
• wordTimestamps: each word's start and end, for karaoke-style captions or search.
• subtitleFormats: ["srt", "vtt"] by default.
• maxMinutesPerFile: stops long files early (they are marked truncated).

Limits

• Links must point to the media file itself. Pages on YouTube, TikTok or Instagram are not files, and the Actor does not extract media from them.
• No speaker labels: the text does not say who spoke.
• base is a small, fast model. It does well on clear speech in widely spoken languages; names, rare words and noisy audio are weaker. Check transcripts that matter against the audio.
• Transcribe only recordings you have the right to process, and do not use the output alone for decisions about people. Both cautions come from the Whisper model card.

Questions about the example are welcome in the repository's Issues.