Web Scrapers in Ruby

Tools used

Ruby
Mechanize (Crawler and Parser) github
MongoDB (Database) github
Mongo Ruby Driver github
Parallel (Multi threading, Multi Processing) github
Typhoeus (Concurrent HTTP requests) github
Delayed Job (ASync Background Job) github

Samples

Plain simple web scraper using mechanize
Storing result in mongo
Progress tracking with mongo
Multi threading with parallel
Concurrent HTTP requests with typhoeus
Background Job with delayed_job