How (not) to integrate Elasticsearch testing with RSpec

In this post I want to share my experience in development of an integration testing tool for a search engine using RSpec framework, which is popular among Ruby developers. My goal was to automate generation of test cases as much as possible. I encountered several problems along the way. I hope this post will prevent readers from making my mistakes.I’ve worked with several companies that use Elasticsearch for data indexing and searching, however none of them have covered their search requests with tests. I think the main reason for lack of tests is the complexity of integration Elasticsearch into a testing environment. The second reason is the enormous amount of work which is required to support documents stubs for necessary test cases. The situation with every project has been almost the same: a single search query is stubbed, then one spec checks that system has sent a request and received mocked data. This spec has a problem, it checks if a query has been sent to Elastic, but it does not care about search results.When I was working in Lobster, we decided to develop a tool which would solve these problems and allow us to cover search engine responses with specs. Lobster is a marketplace for user-generated photos and videos. Photographers connect their accounts from social networks or cloud storages like Instagram, Flickr, Facebook, Dropbox, etc. to the marketplace, then it fetches their photos and videos, indexes them and displays to buyers.