In this article, we show how Puppeteer's advanced capabilities can be used to make our scraper better equipped for handling real world use cases. Namely, we will explore options such as controlling page load behavior, HTTP Authentication, adding extra headers, changing user agent, etc...
In our previous article [https://ujeebu.com/blog/simple-puppeteer-based-scraper-rule-based-extraction/] of the Puppeteer series we implemented a rule-based scraper based on headless Chrome using Puppeteer. We injected our scraping functions into the browser's context (window) then used those to execute scraping scenarios inside the browser. In this article we will try to achieve the same thing, but this time using Puppeteer's methods without injecting functions into the browser's context. Rew
In this article, we show how to scrape any website with a given set of rules using the Puppeteer library.
Web scraping is the process of extracting data from websites. One popular library for web scraping is Puppeteer. Puppeteer is a Node.js library that provides a high-level API to control headless Chrome or Chromium over the DevTools Protocol.
