How I Replaced Heavy Scraping Stacks with Pure Python Stdlib (+ Live Interactive HTML Reports)

2026-11-20来源 dev.toEnglish
How I Replaced Heavy Scraping Stacks with Pure Python Stdlib (+ Live Interactive HTML Reports)

Whenever I need to extract HTML tables or JSON-LD from a static directory, clean up currencies and URLs, and send a stakeholder something better than a raw CSV, pulling in Playwright, BeautifulSoup, and Pandas feels like overkill. For static server-rendered HTML, dragging in hundreds of megabytes of dependencies slows down CI and creates version churn.

So I built DataPipe Kit, a zero-dependency Python 3.9+ CLI and library using only the Python standard library (html.parser, urllib.request, csv, json, re, and html).

DataPipe Kit interactive HTML report preview

One-Command Pipeline

python3 -m datapipe pipeline examples/sample_catalog.html \ --out-dir ./out \ --required Vendor \ --dedupe-by Vendor \ --price-fields "Monthly Price" \ --email-fields "Contact Email" \ --url-fields Website \ --drop-invalid-emails 

From a messy 6-row HTML catalog containing duplicate whitespace entries, malformed emails, and currency strings like $1,250.00, it outputs:

  1. cleaned.csv (UTF-8 BOM so Microsoft Excel opens cleanly without encoding issues)
  2. cleaned.json (typed numeric floats + email validation flags)
  3. report.html (a single-file dark-mode dashboard with live multi-column filtering, numeric/text sorting, KPI cards, and one-click filtered CSV download)

Three Real-World Edge Cases Worth Handling Explicitly

1. Irregular HTML <table> Rows

Real HTML tables routinely omit trailing <td> cells or inject spacer <tr> rows. Padding short rows with "" and skipping blank rows avoids index errors and shifted columns:

def _rows_to_records(rows: list[list[str]]) -> list[dict[str, str]]: if len(rows) < 2: return [] headers = [h if h else f"col_{idx + 1}" for idx, h in enumerate(rows[0])] records: list[dict[str, str]] = [] for row in rows[1:]: record = { header: (row[idx] if idx < len(row) else "") for idx, header in enumerate(headers) } records.append(record) return records 

2. Escaping Inline JSON Against </script> Breakouts

Embedding serialized JSON directly inside a <script> tag breaks if any extracted cell contains </script>. Escaping </ to <\/ in Python and escaping cell entities in the DOM helper closes that XSS vector cleanly:

payload = json.dumps( {"columns": columns, "rows": records, "numericCols": numeric_cols}, ensure_ascii=False, ).replace("</", "<\\/") 

3. Deterministic Currency & URL Normalization

Standardizing strings like "Acme Cloud", "$1,250.00", and "acmecloud.io/" before deduplicating catches duplicates that exact string matching misses, while encoding="utf-8-sig" ensures Excel renders non-ASCII strings properly.

Try It Locally

git clone https://github.com/MarkDonish/datapipe-kit.git cd datapipe-kit python3 -m unittest discover -s tests -v 

All 6 unittest cases pass in under 0.1s with zero third-party installs (CI badge green on GitHub Actions). If you build lightweight Python automation scripts or client data handoffs, feel free to fork or test it out!

Open to custom Python automation, web extraction, and frontend/reporting work — feel free to reach out via GitHub or email marklodpice@gmail.com.

原文:https://www.a8q9.cn/en/article/how-i-replaced-heavy-scraping-stacks-with-pure-python-stdlib-live-interactive-ht-c50531.html

相关文章