Move the hard-coded ALIASES dict into resources/aliases.toml (loaded with
stdlib tomllib), expand it with agencies, international bodies, and country
names, and fix two correctness bugs in the naive replace():
- Whole-word matching so 'united states' no longer collapses inside
'united statesman', and 'inflation rate' no longer mangles 'inflation rates'.
- Longest-phrase-first application so 'president of the united states'
resolves to 'potus' before 'united states' fires.
- Correct a stopword collision: 'united nations' -> 'un' was silently dropped
because 'un' is a stopword; it now maps to 'unitednations'.
Aliases are threaded through normalize_headline -> is_headline ->
_build_headlines -> prepare_headlines and loaded in main.py. Includes unit
tests and README coverage.
services/ssr.py walks JSON-LD and Next.js __NEXT_DATA__ for article-like
objects (headline, or title+url, or name with article signals), recovering
JS-rendered sites without a browser. prepare_headlines favors embedded JSON
over regex link scraping. Forbes is re-added as a live proof.
- services/feeds.py parses RSS 2.0/1.0 and Atom into (title, date) entries
- prepare_headlines auto-detects feeds by content and parses them directly
- strip feeds./rss./moxie. subdomains so feeds collapse to the outlet domain
- add 7 verified RSS feeds (BBC, Guardian, Al Jazeera, WaPo, The Hill, Vox, CNBC)
- get_sources skips blank lines and # comments
- broaden excluded-phrase list (skip links, share/follow, newsletter, utility)
Add resources/excluded_phrases.txt (skip links, privacy/consent, legal
boilerplate) and match it case- and punctuation-insensitively as a
contiguous run of words. Phrases live in a data file so they can be
extended without touching code.
News sites often reject minimal-bot requests; send a full browser header
set (User-Agent, Accept, Sec-Fetch-*, client hints). Accept-Encoding is
left to requests/urllib3 to avoid undecodable brotli responses.
Jared Bell