Fix for Python 3.9+/current arXiv + switch to the official arXiv API

[svemulapati]

Summary

This PR gets sotawhat working again on modern Python and a current arXiv, and makes it resilient to future arXiv changes. The script had stopped working on Python 3.9+ and against arXiv's current site.

Fixes (script was broken on Python 3.9+ / current arXiv)

• Replace removed HTMLParser.unescape() with html.unescape() (removed in Python 3.9).
• Fix garbled paper links and author-name mojibake (proper UTF-8 decoding of the response).
• Auto-configure SSL via certifi, so nltk downloads and arXiv requests work on macOS without the manual "Install Certificates" step.

Improvement: use the official arXiv API

• Switch from scraping arXiv's search HTML to the official, public, no-registration arXiv API, parsing the structured Atom XML with xml.etree.ElementTree. This removes the brittleness that broke the tool whenever arXiv changed its page layout.
• Polite >= 3s delay between paged requests, per arXiv's API terms of use.
• Drop the now-unused six dependency.

Unchanged

• CLI: sotawhat <keyword> [count] (default 5).
• Output format (title, author, date, summary, link).
• The abstract-summarization behavior that prioritizes sentences with numbers / SOTA claims.
• Keyword-based CS-only filtering for plain English words, all-of-arXiv for jargon.

Tested locally across several keywords (transformer, language model, perplexity, GAN, wikitext) plus the no-result path.

🤖 Generated with Claude Code