Free Downloads: Crawlers

License:  All 1 2 | Free

Web Scraper Plus+: Web Spider Edition Screenshot
more

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor.

Build a custom web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own web grabber that can screen scrape the data to a database or Excel has never been easier. The website spider /...

Category: Utilities
Publisher: Velocityscape, License: Shareware, Price: USD $419.00, File Size: 51.3 MB
Platform: Windows



Shareware
more

Manage and track visits by robots, spiders and crawlers to your website.

Manage and track visits by robots, spiders and Crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Maximizes effectiveness with search engines by avoiding over/under submitting. Excellent to improve search engine placements for users of automatic...

Category: Internet / Tools & Utilities
Publisher: Website Management Tools, License: Shareware, Price: USD $129.95, File Size: 3.3 MB
Platform: Windows

Free Large Android Icons Screenshot
more

Free Large Android Icons picture various space creatures, such as robots, crawlers and androids. Not only does the set present the creatures themselves, it also shows their moods and actions.

Free Large Android Icons picture various space creatures, such as robots, Crawlers and androids. Not only does the set present the creatures themselves, it also shows their moods and actions. All the images are created by professional artists and demonstrate bright colors and well-finished edges. This free icon set is a perfect choice for your personal desktop customization. Each...

Category: Desktop Enhancements / Icon Tools
Publisher: large-icons.com, License: Freeware, Price: USD $0.00, File Size: 13.7 MB
Platform: Windows, Mac, Linux, Pocket PC, Other, Android, BlackBerry, iPhone, iPod, iTouch, Palm, Unix, Handheld/Mobile Other, Java, AS/400, OS/2

Advanced Robots.txt Generator Screenshot
more

Create your robots.txt files in seconds and maintain them easilly and effortlesly with the unique utility Advanced Robots.txt Generator. The software supports 300 robots, works in a WYSWIWYG visual way and comes with full documentation and support.

Create your robots.txt files in seconds and maintain them easilly and effortlesly with the unique utility Advanced Robots.txt Generator. Advanced Robots.txt Generator is the most complete and easy to use application that allows you to create, maintain and keep track of your Robots.txt files for all your websites. The following is a list of the features of Advanced Robots.txt Generator. *...

Category: Software Development
Publisher: BasiSoft Solutions, License: Shareware, Price: USD $65.00, File Size: 1.5 MB
Platform: Windows

Sothink DHTML Menu Suite Screenshot
more

Sothink DHTML Menu creates DHTML menu easily with its user-friendly interface and rich templates. It supports creating tab menu. It also supports Google Chrome and integrates well with Dreamweaver, FrontPage, Expression Web and Golive.

Sothink menu builder helps the web designer to create professional DHTML menus easily! It is unnecessary to be familiar with JavaScript or DHTML acknowledge. Just with builder, you can build DHTML drop down, popup or tree-like menus in few minutes. The generated JavaScript menus are compatible with almost all browsers. Seamlessly integrate with Dreamweaver as an extension and FrontPage as an...

Category: Web Authoring / JavaScript
Publisher: SourceTec Software, License: Shareware, Price: USD $99.00, File Size: 16.8 MB
Platform: Windows

Sothink Menu Builder suite Screenshot
more

Build drop-down or tree-like menus in easiest way! The generated JavaScript menus can be compatible with most browsers. It integrates with Dreamweaver and FrontPage seamlessly. The menu's content can be found by Search Engine without difficulty.

Sothink menu builder helps the web designer to create professional DHTML menus easily! It is unnecessary to be familiar with JavaScript or DHTML acknowledge. Just with builder, you can build DHTML drop down, popup or tree-like menus in few minutes. The generated JavaScript menus are compatible with almost all browsers. Seamlessly integrate with Dreamweaver as an extension and FrontPage as an...

Category: Web Authoring / JavaScript
Publisher: SourceTec Software, License: Shareware, Price: USD $99.00, File Size: 16.8 MB
Platform: Windows

Sothink Tree Menu builder Screenshot
more

Build DHTML tree menu easily! Compatible with most browsers included Firefox 2.0 and Opera 9; support Windows Vista. Tree Menu integrates with Dreamweaver, Expression Web, FrontPage and Golive. The generated sitemaps are friendly to Search Engine.

Sothink Tree Menu makes creation tour of JavaScript tree menu simple! Sothink Tree Menu has the features of wide compatibility, high performance, and easy operation. For compatibility, the generated tree menu can be compatible with the most browsers, such as IE 7, Opera 9 and Firefox 2, and Windows Vista included. For performance, Sothink Tree Menu can integrate with Dreamweaver, Expression...

Category: Internet
Publisher: SourceTec Software, License: Shareware, Price: USD $69.95, File Size: 3.8 MB
Platform: Windows

Shareware
more

Eliminate cut and paste! Web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, govt. databases, real estate listings. Our easiest product yet!

Eliminate cut and paste! Web spider / web crawler using web data extraction / screen scraping technology. Use the web extract for web data mining of contact lists, product catalogs, govt. databases, real estate listings. Our easiest product yet! Creating your own web grabber that can screen scrape the web to a database or Excel has never been easier. Web Scraper Lite delivers the data you...

Category: Internet
Publisher: Velocityscape, License: Shareware, Price: USD $89.00, File Size: 50.9 MB
Platform: Windows

XFileSharing Professional Screenshot
more

Run your own file upload site like Rapidshare or MegaUpload with XFileSharing script. XFileSharing script is an advanced solution for creating free file hosting websites.

XFileSharing is an advanced solution for creating professional file sharing services. It can be installed on any virtual/shared or dedicated Linux hosting. You can customize look and feel of your site with basic knowledge of HTML and place banners, adsense or other type advertisement on its pages. You can quickly monetize your service selling premium accounts. Our upload system allow...

Category: Internet / File Sharing
Publisher: SibSoft Ltd., License: Shareware, Price: USD $120.00, File Size: 227.7 KB
Platform: Windows, Mac, Linux, Unix, Source

more

Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once!

Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once! Search engine rivals Google, Yahoo and Microsoft are teaming up to make it easier for Web site owners to make sure their sites get included in the Web indexes, the companies are expected to announce Thursday, Oct 16, 2006. Now Web site owners will be able to go to one place for alerting all...

Category: Web Authoring / Site Management
Publisher: MagneticOne, License: Commercial, Price: USD $39.00, File Size: 29.0 KB
Platform: Windows, Linux, Unix

more

Create your robots.

Create your robots.txt files in seconds and maintain them easilly and effortlesly with the unique utility Advanced Robots.txt Generator. Advanced Robots.txt Generator is the most complete and easy to use application that allows you to create, maintain and keep track of your Robots.txt files for all your websites. The following is a list of the features of Advanced Robots.txt Generator. *...

Category: Web Authoring / HTML Editors
Publisher: BasiSoft Software Solutions, License: Shareware, Price: USD $35.00, File Size: 2.1 MB
Platform: Windows

Freeware
more

Files Crawler is a file search application.

Files Crawler is a file search application. It lets you search tons of files and download music, videos, images, documents and software. Files Crawler is actually performing its search at Google. As Google allows us to search for more than just Web pages, due to its special indexing method of directory listing. Directory listing of Websites, is actually what allows Google to see all sort of...

Category: Internet / Tools & Utilities
Publisher: Speedapps Inc, License: Freeware, Price: USD $0.00, File Size: 747.5 KB
Platform: Windows

more

Run your own file upload site like Rapidshare or MegaUpload with XFileSharing script.

Run your own file upload site like Rapidshare or MegaUpload with XFileSharing script. XFileSharing script is an advanced solution for creating free file hosting websites. It does not require mysql database and can be installed on any virtual/shared or dedicated hosting running Perl (most servers do). You can customize look and feel of your site with just editing HTML files and can place...

Category: Internet / File Sharing
Publisher: SibSoft Ltd., License: Shareware, Price: USD $49.00, File Size: 81.0 KB
Platform: Windows, Mac, Linux, Unix, Source

Freeware
more

ETagFix is an add-on for Microsoft Internet Information Server (IIS) or any web server that supports ISAPI filters, that cuts off part of the etag that IIS produces so that is does not change every time your computer is rebooted.

ETagFix is an add-on for Microsoft Internet Information Server (IIS) or any web server that supports ISAPI filters, that cuts off part of the etag that IIS produces so that is does not change every time your computer is rebooted. Etag is an HTTP header that is produced by IIS to allow web Crawlers and user clients to check against to see if a web page has changed. ETagFix fixes the...

Category: Internet / Tools & Utilities
Publisher: ISAPILabs, License: Freeware, Price: USD $0.00, File Size: 696.3 KB
Platform: Windows

CoffeeCup Sitemapper Screenshot
more

By placing a Sitemap-formatted file on your webserver, you enable the search engine crawlers to find out what pages are present and which have recently changed, and to crawl your site accordingly.

Sitemaps are a great SEO tool. They give you a way to stand up and say, "Hey, search engines! Over here! Check out my site." And sitemaps are also a great usability tool. They give your visitors a secondary navigation option, not to mention a peek at how your website is structured. Yep, sitemaps form a near-perfect intersection of SEO and usability. Unfortunately, they're pretty tedious...

Category: Web Authoring / HTML Editors
Publisher: CoffeeCup Software, License: Shareware, Price: USD $34.00, File Size: 6.4 MB
Platform: Windows

more

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export...

Category: Internet / Communications
Publisher: Changsha aeke Information Technology Co., License: Shareware, Price: USD $0.00, File Size: 1.6 MB
Platform: Windows

obdkit sitemaps generator Screenshot
more

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export...

Category: Internet / Communications
Publisher: obdkit Technology, License: Shareware, Price: USD $0.00, File Size: 1.6 MB
Platform: Windows

Vietspider Web Data Extractor Screenshot
more

The web crawler is a program that automatically traverses the web by downloading the pages and following the links from page to page.

The web crawler is a program that automatically traverses the web by downloading the pages and following the links from page to page. A general purpose of web crawler is to download any web page that can be accessed through the links.

This process is called web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. Web...

Category: Internet / Misc. Plugins
Publisher: VietSpider, License: Freeware, Price: USD $0.00, File Size: 48.8 MB
Platform: Windows, Linux

tradesir sitemaps generator Screenshot
more

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export...

Category: Internet / Download Managers
Publisher: tradesirTechnology, License: Shareware, Price: USD $0.00, File Size: 1.6 MB
Platform: Windows

more

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.

The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and export...

Category: Web Authoring / Site Management
Publisher: Changsha Bowu Information Technology Co., License: Shareware, Price: USD $9.00, File Size: 1.6 MB
Platform: Windows

D-Guard Web Application Firewall Gateway Screenshot
more

Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud.

Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud.
The D-Guard Web Application Firewall is a powerful security tool for Web applications and Web sites.
D-Guard Web Application Firewall is designed to combat all attack types that have been categorized as significant threats,...

Category: Security & Privacy / Access Control
Publisher: D-Guard Technology Co., Ltd., License: Shareware, Price: USD $6999.00, File Size: 4.6 MB
Platform: Windows

D-Guard Web Application Firewall Basic Screenshot
more

Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud.

Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud.
The D-Guard Web Application Firewall is a powerful security tool for Web applications and Web sites.
D-Guard Web Application Firewall is designed to combat all attack types that have been categorized as significant threats,...

Category: Security & Privacy / Access Control
Publisher: D-Guard Technology Co., Ltd., License: Shareware, Price: USD $399.00, File Size: 4.6 MB
Platform: Windows

InnerPeace Webmaster ToolKit - Assorted Utilities Screenshot
more

The InnerPeace WebMaster Toolkit is a collection of essensial tools assembeled and made by the Webmasters of InnerPeace.

The InnerPeace WebMaster Toolkit is a collection of essensial tools assembeled and made by the Webmasters of InnerPeace. Some of these essensial tools are not available anywhere else. Includes a fast, easy 4 hexillion color combo picker, PageJumper web page forwarder, EScrambler to scrambler your email addresses and hide them from spamcrawlers, plus some excellent articles. Colors on web pages...

Category: Utilities / Misc. Utilities
Publisher: innerpeace.org, License: Freeware, Price: USD $0.00, File Size: 370.0 KB
Platform: Windows, Linux, Mac, Palm

XML Sitemap for CRE Loaded - CRE Loaded Screenshot
more

Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once!

Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once!

Search engine rivals Google, Yahoo and Microsoft are teaming up to make it easier for Web site owners to make sure their sites get included in the Web indexes, the companies are expected to announce Thursday, Oct 16, 2006.

Now Web site owners will be able to go to one place for...

Category: Web Authoring / Web Design Utilities
Publisher: MagneticOne, License: Commercial, Price: USD $39.00, File Size: 30.0 KB
Platform: Windows, Linux

Shareware
more

SearchBlox is an out-of-the-box Enterprise Search Solution built on top of Apache Lucene.

SearchBlox is an out-of-the-box Enterprise Search Solution built on top of Apache Lucene. It is fast to deploy, easy to manage and available for both on-premise and cloud deployment.

Best of all, it is free. No limitations. No restrictions.
SearchBlox provides Crawlers for Filesystems, HTTP/HTTPS and RSS Feeds that work out-of-the-box. The HTTP/HTTPS crawler can index...

Category: Internet / Web Search Utilities
Publisher: SearchBlox Software, Inc., License: Shareware, Price: USD $0.00, File Size: 0
Platform: Windows

blank

License:  All 1 2 | Free