Also known as a crawler, a bot or a spider, the search engine robot is a program crawls through your website to index its pages. This allows search engines to identify relevant pages and serve them to ...