Free Downloads: Web Crawlers
ETagFix is an add-on for Microsoft Internet Information Server (IIS) or any Web server that supports ISAPI filters, that cuts off part of the etag that IIS produces so that is does not change every time your computer is rebooted. Etag is an HTTP header that is produced by IIS to allow Web Crawlers and user clients to check against to see if a Web page has changed....
Category: Internet / Tools & Utilities
Publisher: ISAPILabs, License: Freeware, Price: USD $0.00, File Size: 696.3 KB
Platform: Windows
The Web crawler is a program that automatically traverses the Web by downloading the pages and following the links from page to page. A general purpose of Web crawler is to download any Web page that can be accessed through the links.
This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of...
Category: Internet / Misc. Plugins
Publisher: VietSpider, License: Freeware, Price: USD $0.00, File Size: 48.8 MB
Platform: Windows, Linux
Manage and track visits by robots, spiders and crawlers to your website.
Manage and track visits by robots, spiders and Crawlers to your website. Identify which spiders visited, when they visited, and what pages they indexed. Direct spiders to your most important pages, and exclude them from non-public pages. Maximizes effectiveness with search engines by avoiding over/under submitting. Excellent to improve search engine placements for users of automatic...
Category: Internet / Tools & Utilities
Publisher: Website Management Tools, License: Shareware, Price: USD $129.95, File Size: 3.3 MB
Platform: Windows
Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once!
Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once! Search engine rivals Google, Yahoo and Microsoft are teaming up to make it easier for Web site owners to make sure their sites get included in the Web indexes, the companies are expected to announce Thursday, Oct 16, 2006. Now Web site owners will be able to go to one place...
Category: Web Authoring / Site Management
Publisher: MagneticOne, License: Commercial, Price: USD $39.00, File Size: 29.0 KB
Platform: Windows, Linux, Unix
Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once!
Generates XML Sitemap, REQUIRED for search engine optimization for Google, MSN, Yahoo - all at once!
Search engine rivals Google, Yahoo and Microsoft are teaming up to make it easier for Web site owners to make sure their sites get included in the Web indexes, the companies are expected to announce Thursday, Oct 16, 2006.
Now Web site owners will be able to...
Category: Web Authoring / Web Design Utilities
Publisher: MagneticOne, License: Commercial, Price: USD $39.00, File Size: 30.0 KB
Platform: Windows, Linux
Build a custom Web spider / Web crawler using Web data extraction / screen scraping technology. Use the Web extract for Web data mining of contact lists, product catalogs, government databases, real estate listings, or build a custom email extractor. Creating your own Web grabber that can screen scrape the data to a database or Excel has never been...
Category: Utilities
Publisher: Velocityscape, License: Shareware, Price: USD $419.00, File Size: 51.3 MB
Platform: Windows
Freeware
Free Large Android Icons picture various space creatures, such as robots, Crawlers and androids. Not only does the set present the creatures themselves, it also shows their moods and actions. All the images are created by professional artists and demonstrate bright colors and well-finished edges. This free icon set is a perfect choice for your personal desktop customization. Each...
Category: Desktop Enhancements / Icon Tools
Publisher: large-icons.com, License: Freeware, Price: USD $0.00, File Size: 13.7 MB
Platform: Windows, Mac, Linux,
This program lets you to substitute URI for symbol codes of the ASCII table.
This program lets you to substitute URI for symbol codes of the ASCII table. It's very useful to prevent scanning of Web pages by the Crawlers gathering e-mail addresses for broadcasting spam messages on the next step. Using this software you can protect your mailbox from tons of junk mail, and will not be afraid of publishing contacts on your home page or a company's...
Category: Security & Privacy / Encrypting
Publisher: Loonies Software, License: Freeware, Price: USD $0.00, File Size: 63.3 KB
Platform: Windows
Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud.
The D-Guard Web Application Firewall is a powerful security tool for Web applications and Web sites.
D-Guard Web Application Firewall is designed to combat all attack types that have been categorized as...
Category: Security & Privacy / Access Control
Publisher: D-Guard Technology Co., Ltd., License: Shareware, Price: USD $399.00, File Size: 4.6 MB
Platform: Windows
Online Web-based applications are increasingly at risk from professional hackers who target such applications in order to commit data theft or fraud.
The D-Guard Web Application Firewall is a powerful security tool for Web applications and Web sites.
D-Guard Web Application Firewall is designed to combat all attack types that have been categorized as...
Category: Security & Privacy / Access Control
Publisher: D-Guard Technology Co., Ltd., License: Shareware, Price: USD $6999.00, File Size: 4.6 MB
Platform: Windows
Eliminate cut and paste! Web spider / Web crawler using Web data extraction / screen scraping technology. Use the Web extract for Web data mining of contact lists, product catalogs, govt. databases, real estate listings. Our easiest product yet! Creating your own Web grabber that can screen scrape the Web to a database or Excel has never been...
Category: Internet
Publisher: Velocityscape, License: Shareware, Price: USD $89.00, File Size: 50.9 MB
Platform: Windows
The InnerPeace WebMaster Toolkit is a collection of essensial tools assembeled and made by the Webmasters of InnerPeace. Some of these essensial tools are not available anywhere else. Includes a fast, easy 4 hexillion color combo picker, PageJumper Web page forwarder, EScrambler to scrambler your email addresses and hide them from spamcrawlers, plus some excellent articles. Colors on...
Category: Utilities / Misc. Utilities
Publisher: innerpeace.org, License: Freeware, Price: USD $0.00, File Size: 370.0 KB
Platform: Windows, Linux, Mac, Palm
Freeware
Octoparse is a free client-side Windows Web scraping software that turns unstructured or semi-structured data from websites into structured data sets, no coding necessary. It's an easy-to-use Web scraping tools that collects data from the Web. Crawlers run in Octoparse are determined by the extraction rules configured. The extraction rule would tell Octoparse:...
Category: Internet / Monitoring
Publisher: Octopus Data Inc., License: Freeware, Price: USD $0.00, File Size: 54.0 MB
Platform: Windows
Files Crawler is a file search application.
Files Crawler is a file search application. It lets you search tons of files and download music, videos, images, documents and software. Files Crawler is actually performing its search at Google. As Google allows us to search for more than just Web pages, due to its special indexing method of directory listing. Directory listing of Websites, is actually what allows Google to see all sort...
Category: Internet / Tools & Utilities
Publisher: Speedapps Inc, License: Freeware, Price: USD $0.00, File Size: 747.5 KB
Platform: Windows
The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling Web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and...
Category: Internet / Communications
Publisher: Changsha aeke Information Technology Co., License: Shareware, Price: USD $0.00, File Size: 1.6 MB
Platform: Windows
The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling Web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and...
Category: Internet / Communications
Publisher: obdkit Technology, License: Shareware, Price: USD $0.00, File Size: 1.6 MB
Platform: Windows
The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling Web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and...
Category: Internet / Download Managers
Publisher: tradesirTechnology, License: Shareware, Price: USD $0.00, File Size: 1.6 MB
Platform: Windows
The main features:
Automatically generated sitemaps, With yahoo, google and other large-scale search engine standards.
Simulation crawling Web Crawlers,can check out the incorrect link.
Set up a free multi-threaded, to generate extremely fast.
Manual support Web site editor, artificial setting parameters such as frequency of updates.
To support the import and...
Category: Web Authoring / Site Management
Publisher: Changsha Bowu Information Technology Co., License: Shareware, Price: USD $9.00, File Size: 1.6 MB
Platform: Windows
SearchBlox is an out-of-the-box Enterprise Search Solution built on top of Apache Lucene.
SearchBlox is an out-of-the-box Enterprise Search Solution built on top of Apache Lucene. It is fast to deploy, easy to manage and available for both on-premise and cloud deployment.
Best of all, it is free. No limitations. No restrictions.
SearchBlox provides Crawlers for Filesystems, HTTP/HTTPS and RSS Feeds that work out-of-the-box. The HTTP/HTTPS crawler can index...
Category: Internet / Web Search Utilities
Publisher: SearchBlox Software, Inc., License: Shareware, Price: USD $0.00, File Size: 0
Platform: Windows
Create your robots.txt files in seconds and maintain them easilly and effortlesly with the unique utility Advanced Robots.txt Generator. Advanced Robots.txt Generator is the most complete and easy to use application that allows you to create, maintain and keep track of your Robots.txt files for all your websites. The following is a list of the features of Advanced Robots.txt Generator. *...
Category: Web Authoring / HTML Editors
Publisher: BasiSoft Software Solutions, License: Shareware, Price: USD $35.00, File Size: 2.1 MB
Platform: Windows
Create your robots.txt files in seconds and maintain them easilly and effortlesly with the unique utility Advanced Robots.txt Generator. Advanced Robots.txt Generator is the most complete and easy to use application that allows you to create, maintain and keep track of your Robots.txt files for all your websites. The following is a list of the features of Advanced Robots.txt Generator. *...
Category: Software Development
Publisher: BasiSoft Solutions, License: Shareware, Price: USD $65.00, File Size: 1.5 MB
Platform: Windows
Sothink Tree Menu makes creation tour of JavaScript tree menu simple! Sothink Tree Menu has the features of wide compatibility, high performance, and easy operation. For compatibility, the generated tree menu can be compatible with the most browsers, such as IE 7, Opera 9 and Firefox 2, and Windows Vista included. For performance, Sothink Tree Menu can integrate with Dreamweaver, Expression...
Category: Internet
Publisher: SourceTec Software, License: Shareware, Price: USD $69.95, File Size: 3.8 MB
Platform: Windows
Shareware
Sothink menu builder helps the Web designer to create professional DHTML menus easily! It is unnecessary to be familiar with JavaScript or DHTML acknowledge. Just with builder, you can build DHTML drop down, popup or tree-like menus in few minutes. The generated JavaScript menus are compatible with almost all browsers. Seamlessly integrate with Dreamweaver as an extension and FrontPage...
Category: Web Authoring / JavaScript
Publisher: SourceTec Software, License: Shareware, Price: USD $99.00, File Size: 16.8 MB
Platform: Windows
Shareware
Sothink menu builder helps the Web designer to create professional DHTML menus easily! It is unnecessary to be familiar with JavaScript or DHTML acknowledge. Just with builder, you can build DHTML drop down, popup or tree-like menus in few minutes. The generated JavaScript menus are compatible with almost all browsers. Seamlessly integrate with Dreamweaver as an extension and FrontPage...
Category: Web Authoring / JavaScript
Publisher: SourceTec Software, License: Shareware, Price: USD $99.00, File Size: 16.8 MB
Platform: Windows
Lazy Web Search is a popular web tool to search the web by hot key.
Lazy Web Search is a popular Web tool to search the Web by hot key. With Lazy Web Search you can search the Web from Microsoft Word, Excel, Internet Explorer, and even from Notepad. You don't have to type keywords into search box anymore; just mark an appropriate text and press Ctrl + F12 key combination to see Web search results. Enjoy!
Category: Internet / Tools & Utilities
Publisher: Scorpio Software, License: Freeware, Price: USD $0.00, File Size: 348.0 KB
Platform: Windows
 |