Firecrawl

Firecrawl turns websites into model-ready text. Point it at a URL and it renders the page, strips the navigation and boilerplate, and returns clean markdown; point it at a domain and it will crawl and do the same across the site. It handles JavaScript rendering, which is where most naive scrapers fall over.

We track it for the ingestion end of retrieval work — building a knowledge base from documentation, pulling structured data out of pages that offer no API. The reason it earns a place is that the boring parts are handled: rendering, rate limits, retries, and stripping the furniture off a page so a model sees the content rather than the menu.

Jina Reader is the lightweight alternative and free for modest use — put a URL behind their endpoint and get markdown back, with no crawl orchestration. Apify is the heavier option, a full scraping platform for jobs that need custom logic and scheduling. Playwright or Puppeteer are what you'd reach for when the target is genuinely awkward and you need to drive a browser yourself. Firecrawl's position is the middle: more capable than a reader endpoint, far less work than building your own crawler.