Web Scraping

High-level, asynchronous web crawling & scraping framework with spiders, pipelines, middlewares, and selectors.

Python library for pulling data out of HTML and XML files; pairs well with requests, lxml, or html.parser.

Microsoft's modern browser automation library for Chromium, Firefox, and WebKit — async-first, with auto-waiting and powerful selectors.

The de facto standard for browser automation — drives any major browser via WebDriver protocol, mature ecosystem with grid for parallel execution.

LLM-friendly web crawler and scraper that returns clean markdown/JSON optimized for RAG and LLM ingestion — async-first with browser automation.