The Free Press Gave Me 9% of a Paywalled Post. Lenny's Gave 88%. Both Rows Carry the Same Flag

I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them reads Substack. Substack serves its own archive pages as JSON, so a publication's whole back catalogue is a few HTTP calls away with no browser and no login — which makes it a tempting source for anything content-shaped: competitor tracking, topic research, RAG corpora.

This week I went to do the obvious thing with it. Pull a few newsletters, pull the body text, and compute something boring like words per like. The arithmetic fell apart on the first sort, and the reason is a single integer field that means two different things depending on who published the post.

The run

Four publications, 25 newest posts each, full text switched on:

{ "publications": [ "https://www.lennysnewsletter.com", "noahpinion.substack.com", "https://www.astralcodexten.com", "https://www.thefp.com" ], "maxPostsPerPublication": 25, "includeContent": true } 

100 post rows, 4 publication rows, 7 m 50 s, charged $0.09025. The split: 51 free posts, 49 paid, though that split is wildly uneven by publication — 22 of The Free Press's 25 newest posts are paywalled, against 1 of 25 at Astral Codex Ten.

Every row carries Substack's own wordCount (its wordcount field, verbatim) and, with the switch on, contentHtml plus a plain-text contentText. I counted the words in contentText myself with len(re.findall(r"[A-Za-z0-9'-]+", text)) and divided.

Free posts: the number is honest

Median of 1.005 on 51 posts is as close to "the same number" as two different tokenisers get. The 1.298 outlier is interesting though, and it is not a scraping failure.

That post is an Astral Codex Ten book review of Lee Kuan Yew's memoirs: wordCount 5,683, my count 7,375. Its HTML contains 72 <div class="footnote"> blocks. Strip those and re-count: 5,652 against a claimed 5,683 — a 0.5% gap, which is just my regex and Substack's disagreeing about hyphens.

So: Substack's wordcount excludes footnotes. Seventeen of my 51 free posts read more than 2% long, and sixteen of those were footnote-carrying Astral Codex Ten and Noahpinion pieces. "Storm Of Steel" (36 footnotes): 7,559 claimed, 8,625 counted, 7,551 with footnotes removed. "Why Donald Trump won in 2024" (34 footnotes): 12,818 claimed, 13,454 counted, 12,769 without. If you are building a reading-time estimate or chunking for RAG, that is a real 8–30% of the text living outside the official length.

Paid posts: the number describes a document you don't have

Paid posts come back with every piece of metadata intact — title, date, likes, comments, restacks, wordCount — and the body you get is the publisher's free preview, correctly flagged as contentIsPreview: true and, sensibly, not charged for.

What no field tells you is how much of the post that preview is. On my 49 paid posts:

Half the corpus, and the halving is not uniform. Grouped by publication it stops looking like a platform behaviour and starts looking like an editorial decision:

Concretely: How to turn your AI into a world-class design partner claims 4,604 words at Lenny's and handed me 4,028 of them. Meet the 'Election Defenders' Preparing for a Contested Election claims 1,751 words at The Free Press and handed me 167. Both rows say isPaid: true, contentIsPreview: true. Nothing in the data distinguishes "we show you most of it" from "we show you the first paragraph."

Two things make this easy to miss:

The previews do not look truncated. Only 2 of 49 paid bodies end in an ellipsis, and both are Free Press posts. Lenny's previews just stop — my sample's cut on How to tell if a startup has a moat lands mid-argument on "...but many still fail to become good businesses." If your pipeline's heuristic for "is this complete" is a trailing …, it will pass 47 of 49 truncated documents.

contentIsPreview is not a paywall flag. One paid post, Astral Codex Ten's Hidden Open Thread 454.5, returned contentHtml: null — nothing at all. That makes contentIsPreview false, because there is no preview to flag. Filter on isPaid, not on the preview flag, or that row quietly joins your "full text" bucket with an empty body.

The practical rule I now use: wordCount is a property of the post; contentText is a property of what Substack is willing to show an anonymous reader. Never divide one by the other across a mixed free/paid sample, and never put paid bodies in the same corpus as free ones without recording the share:

import re WORD = re.compile(r"[A-Za-z0-9'\-]+") for r in (x for x in items if x["type"] == "post"): got = len(WORD.findall(r.get("contentText") or "")) share = got / r["wordCount"] if r.get("wordCount") else None r["textShare"] = share # 1.0-ish = whole post; 0.09 = first paragraph  r["usableText"] = (not r["isPaid"]) and got > 0 

Two smaller traps in the same dataset

The publication row's url is whatever you typed. I passed noahpinion.substack.com; the publication row came back with url: "https://noahpinion.substack.com" while all 25 of its post URLs are on www.noahpinion.blog. Join posts to publications on the publication name or on customDomain, never on the host of url. (The Free Press is the same trap from the other end: its subdomain is bariweiss, not thefp — the brand was renamed, the subdomain was not.)

subscriberTier is empty more often than you'd guess. Substack publishes a band, not a count — "Millions of subscribers" for Lenny's, "Hundreds of thousands" for Noahpinion — and subscribersOrderOfMagnitude turns that into 1000000 and 100000 respectively. But Astral Codex Ten and The Free Press returned null for both fields. Two of four. If you are sizing a niche, expect a coverage hole, not a number for every row; plans was filled for all four ($10/month and $100/year at three of them, $20/$200 at Lenny's).

Calling it

curl -X POST "https://api.apify.com/v2/acts/lergassy~substack-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"publications":["noahpinion.substack.com"],"maxPostsPerPublication":2}' 

That exact call, this morning: 22 seconds, 4 rows, publication ×1 + post ×2 = $0.0029. The last row has type: "notice" and is a free plain-English note about what you were charged — it is not data, filter it out. The Actor is Substack Scraper.

One operational note from the logs: a single request to noahpinion.substack.com died with tls handshake eof and succeeded on retry from a different address. *.substack.com is pickier than the custom domains — it answers 403 to a Chrome-impersonating client from every address I have, while a plain client gets a 200 and the redirect. If you write your own Substack client and it works on your laptop but 403s in CI, that asymmetry is what you are hitting, and the fix is less browser-like, not more.

When this is the wrong tool

If you need the body of paid posts, nothing here helps and nothing should — the median paid preview in my sample was a third of the article, and no proxy changes that. If you need subscriber counts rather than bands, Substack does not publish them. And if you want comments or Notes, this reads posts and publications only. Where it does earn its keep is the metadata layer: 100 posts across four publications with likes, restacks, word counts, authors, tags and paywall status, in under eight minutes for nine cents — the archive endpoint hands all of that over in one call per 20 posts, whether you ask for the text or not.