Web Scraping with Ruby

This post will cover main tools and techniques for web scraping in Ruby. We start with an introduction to building a web scraper using common Ruby HTTP clients and parsing the response. This approach to web scraping has, however, its limitations and can come with a fair dose of frustration. Not to mention, as manageable as it is to scrape static pages, these tools fail when it comes to dealing with Single Page Applications, the content of which is built with JavaScript. As an answer to that, we will propose using a complete web scraping framework. This article assumes that the reader is familiar with fundamentals of Ruby and of how the Internet works.