How to pull a Substack newsletter's posts as Markdown
You want a newsletter's archive as Markdown files, for offline reading, a search index or a knowledge base. Every Substack publication serves its archive and its posts as JSON, on a name.substack.com address and on custom domains, and free posts need no login.
The archive endpoint
GET https://<publication>/api/v1/archive?sort=new&limit=25 returns {"posts": [...], "nextCursor": "..."}. Each post has id, slug, title, subtitle, post_date, audience, type, wordcount, canonical_url and reaction and comment counts. The body is not in the listing: body_html is null there.
Two quirks. limit=25 works and limit=50 returned HTTP 400 today. offset is rejected with 400 as well, so you page with the opaque nextCursor string, passed back as &cursor=. Replay it as given. The last page has nextCursor: null.
import json, re, time, urllib.request from html.parser import HTMLParser UA = {"User-Agent": "substack-export-example/1.0 (personal script)"} def get(url): with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r: return json.load(r) def archive(pub): """Yield post metadata, newest first, following the server's cursor.""" cursor = None while True: url = f"https://{pub}/api/v1/archive?sort=new&limit=25" if cursor: url += "&cursor=" + cursor page = get(url) yield from page["posts"] cursor = page.get("nextCursor") if not cursor: return time.sleep(1) # stay near one request per second for i, post in enumerate(archive("importai.substack.com")): if i < 3: print(post["post_date"][:10], post["audience"], post["wordcount"], post["slug"]) print(i + 1, "posts") Output:
2026-11-09 everyone 2622 import-ai-480-self-sustaining-space 2026-11-02 everyone 2425 import-ai-479-spacewar-low-latency 2026-10-26 everyone 2197 import-ai-478-math-for-alignment 163 posts Fetch a post and convert it
GET https://<publication>/api/v1/posts/<slug> returns one post with body_html. Substack's HTML includes subscribe forms, SVG icons and buttons, which you want to drop. This converter handles headings, emphasis, links, images, lists, quotes, code and fenced code, using only the standard library. Save it as to_markdown.py.
import re from html.parser import HTMLParser SKIP = {"form", "svg", "button", "script", "style"} # subscribe widgets, icons START = {"strong": "**", "b": "**", "em": "_", "i": "_", "li": "\n- ", "blockquote": "\n\n> ", "hr": "\n\n---\n\n", "pre": "\n\n``` \n", **{f"h{n}": "\n\n" + "#" * n + " " for n in range(1, 7)}} END = {"strong": "**", "b": "**", "em": "_", "i": "_", "p": "\n\n", "pre": "\n ```\n\n", **{f"h{n}": "\n\n" for n in range(1, 7)}} class Markdown(HTMLParser): def __init__(self): super().__init__(convert_charrefs=True) self.out, self.skip, self.pre, self.href = [], 0, False, None def handle_starttag(self, tag, attrs): a = dict(attrs) self.skip += tag in SKIP if self.skip: return self.pre = self.pre or tag == "pre" if tag == "a": self.href = a.get("href"); self.out.append("[") elif tag == "img" and a.get("src"): self.out.append(f'') elif tag == "code" and not self.pre: self.out.append("`") else: self.out.append(START.get(tag, "")) def handle_endtag(self, tag): self.skip -= tag in SKIP if self.skip: return if tag == "pre": self.pre = False self.out.append(f"]({self.href})" if tag == "a" else "`" if tag == "code" and not self.pre else END.get(tag, "")) def handle_data(self, data): if not self.skip: self.out.append(data if self.pre else re.sub(r"\s+", " ", data)) def to_markdown(html): md = Markdown(); md.feed(html) return re.sub(r"\n{3,}", "\n\n", "".join(md.out)).strip() Then a small script to fetch one post:
import json, sys, urllib.request from to_markdown import to_markdown pub, slug = sys.argv[1], sys.argv[2] # e.g. importai.substack.com import-ai-480-self-sustaining-space req = urllib.request.Request(f"https://{pub}/api/v1/posts/{slug}", headers={"User-Agent": "substack-export-example/1.0 (personal script)"}) post = json.load(urllib.request.urlopen(req, timeout=30)) text = to_markdown(post["body_html"] or "") print(f'{post["title"]} | audience={post["audience"]} | wordcount={post["wordcount"]}') print(text[:500]) $ python get_post.py importai.substack.com import-ai-480-self-sustaining-space Import AI 480: Self-sustaining space economies; AI for science progress, ... | audience=everyone | wordcount=2622 Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. ... [Subscribe now](https://importai.substack.com/subscribe?) The converter is deliberately small. Ordered lists become bullets, <br> is dropped, and a quote with several paragraphs only gets > on its first line. A library such as markdownify or html2text will do better if you need fidelity.
Paywalled posts
audience is everyone for free posts. Anything else, such as only_subscribers or only_paid, is gated. For a gated post the endpoint returns only the public preview. I fetched a paid post from a newsletter that reports wordcount=6286; the API returned roughly 2,850 words of it, the part Substack shows to anyone.
... | audience=only_paid | wordcount=6286 Paid content is not accessible through this endpoint, and you should not try to get it another way: no borrowed cookies, no logging in on someone else's behalf. Check audience == "everyone" before you treat a body as a complete post, or a search index will fill up with half-articles. These are other people's writing, so keep archives for your own use unless the author has agreed to more.
Limits and gotchas
- Rate limit. Roughly two requests a second sustained is too fast. During my testing, two parallel workers with a half-second delay drew a burst of HTTP 429 after about 100 requests. One request per second was clean, including the 7 archive pages above, so the loop sleeps a second and you should back off on a 429.
- Hidden posts. Publishers can hide sections, and often their podcast, from the main archive.
/api/v1/archive?section_id=<id>lists a section, and/sitemap.xmllists every public post URL with alastmod. On one large newsletter the main archive held 485 posts and the sitemap 1,296. Compare the sitemap count with what you pulled. - Not everything on a custom domain is still Substack. A site that moved away answers with a plain 404, not a Substack error.
- Some endpoints refuse you.
/api/v1/publicationreturned 403 on every newsletter I tried, and the globalsubstack.comsearch and category pages do not answer anonymous API clients. - Public data only. Skip comments and anything about readers. Keep the request rate low and follow the publication's terms.
If you don't want to maintain this yourself
Disclosure: I'm the developer of an Apify Actor, Substack Scraper, published under Cedarfetch Labs. It does the paging and conversion above and adds Markdown with footnotes and embeds handled, rows marked isPaywalled and contentAvailable: "preview" so previews never pass for full posts, an option to include hidden sections and sitemap-only posts, and a new-posts feed through change tracking. It never logs in and never gets past a paywall.
It is pay per event: $0.001 per post row, body included, plus $0.00005 per run start, as shown on the listing on 2026-11-11. Check the Pricing tab for current figures. Listing: https://apify.com/cedarfetch/substack-scraper
Runnable Python, Node.js and cURL examples for the Actor: https://github.com/cedarfetch/substack-scraper-examples
Not affiliated with Substack.
原文:https://www.a8q9.cn/en/article/how-to-pull-a-substack-newsletter-s-posts-as-markdown-d549cf.html