Stealthly Browsing and Scraping with Ferrum

I've been using Ferrum to do a lot of web-scraping lately (I'm building a link checker tool), and I wanted to share some tips and best-practices I've stumbled on developed. Ferrum is a headless browser driver, similar to Playwright and Puppeteer, which you can use to automatically visit, interact with, and scrape data from websites. Ferrum has been gaining popularity lately in the Ruby on Rails community for being fast and Ruby-native. Lately, I needed to do some web-scraping for Affimon, so I decided to give Ferrum a go. In this article, I share everything I've learned about stealthy scraping with Ferrum — how to avoid basic blocks, preserve bandwidth, rotate user agents and integrate with proxies. I've also included a few sites in this article's appendix to test your bot detection.