Céline Debled reposted
A fetch to the NeurIPS proceedings page gets zero papers, because the list is built by JavaScript. 🫥 From Python, Lightpanda reads all 5,858 out of the live DOM. On a 100 quote scrape: > 3s and 35 MB on Lightpanda > 6s and 1.2 GB for Selenium + headless Chrome lightpanda.io/blog/posts/l...
lightpanda.ioLightpanda for Python: a browser in one pip install - Blog | Lightpandapip install lightpanda gives you a headless browser and its Python bindings in one command, browser included. Scrape JavaScript-rendered pages and hand the data straight to pandas.