Sometimes called a bot, robot or spider. Search engines work by indexing all available web pages and scoring them based on a number of factors to gauge their trustworthiness or popularity. Web crawlers are automatic routines that travel the web using links and carry out evaluations of each individual website. Google’s PageRank Web Crawler is possibly the most famous.
« Back to Glossary Index
« Back to Glossary Index