Spider
A spider (also known as a crawler) is used by a search engine to identify web pages in order to be cataloged in the database of the search engine (web site designers can also submit web pages to be cataloged in a search engine). Spiders therefore help search engines to keep up with the evolution of web content by crawling through the web seeking new sites. To control spiders in terms of cataloging a web site, a designer can include a robots.txt file that instructs a spider to exclude pages that are not to be cataloged.
Maximizing Business Potential
