Move the hard-coded ALIASES dict into resources/aliases.toml (loaded with
stdlib tomllib), expand it with agencies, international bodies, and country
names, and fix two correctness bugs in the naive replace():
- Whole-word matching so 'united states' no longer collapses inside
'united statesman', and 'inflation rate' no longer mangles 'inflation rates'.
- Longest-phrase-first application so 'president of the united states'
resolves to 'potus' before 'united states' fires.
- Correct a stopword collision: 'united nations' -> 'un' was silently dropped
because 'un' is a stopword; it now maps to 'unitednations'.
Aliases are threaded through normalize_headline -> is_headline ->
_build_headlines -> prepare_headlines and loaded in main.py. Includes unit
tests and README coverage.
services/ssr.py walks JSON-LD and Next.js __NEXT_DATA__ for article-like
objects (headline, or title+url, or name with article signals), recovering
JS-rendered sites without a browser. prepare_headlines favors embedded JSON
over regex link scraping. Forbes is re-added as a live proof.
- Lower `SIMILARITY_THRESHOLD` to 0.68 and raise `MIN_SOURCES` to 3 for stricter story clustering.
- Expand `sources.txt` with new international and regional RSS feeds, including RT, The Jakarta Post, Philstar, and others.
- Add a new "Miscellaneous" section for diverse investigatory sources like The Intercept, OCCRP, and Bellingcat.
Add English-language international feeds: DW, Le Monde (English ed.),
France24, Euronews, Der Spiegel, El Pais (EN), The Japan Times, SCMP,
ABC News Australia, and CBC. Nikkei offers no clean public RSS, so Japan
Times + SCMP cover Japan/Asia instead.
Replace JS-heavy/paywalled HTML sources with verified RSS feeds (CNN,
Fox, NBC, NPR, CBS, NYT, Time, Axios). Drop Reuters, AP, Forbes, MSN and
US News, which offer no public/working RSS and are not reliably scrapable.
- services/feeds.py parses RSS 2.0/1.0 and Atom into (title, date) entries
- prepare_headlines auto-detects feeds by content and parses them directly
- strip feeds./rss./moxie. subdomains so feeds collapse to the outlet domain
- add 7 verified RSS feeds (BBC, Guardian, Al Jazeera, WaPo, The Hill, Vox, CNBC)
- get_sources skips blank lines and # comments
- broaden excluded-phrase list (skip links, share/follow, newsletter, utility)
Add resources/excluded_phrases.txt (skip links, privacy/consent, legal
boilerplate) and match it case- and punctuation-insensitively as a
contiguous run of words. Phrases live in a data file so they can be
extended without touching code.
News sites often reject minimal-bot requests; send a full browser header
set (User-Agent, Accept, Sec-Fetch-*, client hints). Accept-Encoding is
left to requests/urllib3 to avoid undecodable brotli responses.
Jared Bell