awesome-crawler

List of Some Crawler!

Content:

• Java
• Python
• Cpluplus
• C Sharp
• C
• PHP

###Java Heritrix Java Linux

Nutch Java Cross-platform

Norconex HTTP Collector Java Cross-platform

WebSPHINX Java Cross-platform

WebLech Java Cross-platform

Arale Java Cross-platform

JSpider Java Cross-platform

HyperSpider Java Cross-platform

Arachnid Java Cross-platform

Spindle Java Cross-platform

Spider Java Cross-platform

LARM Java Cross-platform

Metis Java Cross-platform

SimpleSpider Java Cross-platform

Grunk Java Cross-platform

CAPEK Java Cross-platform

Aperture Java Cross-platform

Smart and Simple Web Crawler Java Cross-platform

Web Harvest Java Cross-platform

YaCy Java Cross-platform

Bixo Java Cross-platform

crawler4j Java Cross-platform

Distributed Web Crawler C, Java, Python Cross-platform

iCrawler Java Cross-platform

pycreep Java Cross-platform

Andjing Java

WebEater Java Cross-platform

JoBo Java Cross-platform

Hounder Java Cross-platform

###Python Scrapy Python Cross-platform

###Cpluplus

DataparkSearch C++ Cross-platform

ht://Dig C++ Unix

HTTrack C/C++ Cross-platform

ICDL Crawler C++ Cross-platform

Aspseek C++ Linux

Hyper Estraier C/C++ Cross-platform

Xapian C++ Cross-platform

Crawwwler C++ Java

Opese C++ Linux

###C

GNU Wget C Linux

mnoGoSearch C Windows

Pavuk C Lunix

GRUB C#, C, Python, Perl Cross-platform

Open Source Server C/C++, Java PHP Cross-platform

###C Sharp Arachnode.net C# Windows

Ccrawler C# Windows

OpenWebSpider C#, PHP Cross-platform

###PHP

Sphider PHP Cross-platform

PHP-Crawler PHP Cross-platform