The index sitemap should contain all the links to the various XML sitemaps. It may look like: code2 You must then upload the file to Search Console to allow Google to re-crawl the subpages. Advertising Continue reading below If you have a lot of videos and images on your website, you should also check Universal Search indexing by creating separate sitemaps for images and videos. The structure of an XML sitemap for media files is similar to that of the normal sitemap. Point In many cases, you want your website to be crawled again as soon as possible after making several changes. Google Search Console helps in such cases. Go to the affected website and immediately submit it to the Google index. This feature is limited to 500 URLs per month for each website. Explore like Google 2. Use the crawl budget The Google crawler is a computer program designed to follow links, crawl URLs, and then interpret, classify, and index content.
To do this, the bot has a limited fax list crawl budget. The number of crawled and indexed pages depends on the page rank of the respective website, as well as how easily the bot can follow the links on the website. Advertising Continue reading below An optimized website architecture will greatly facilitate the task of the bot. In particular, flat hierarchies help ensure that the bot accesses all available web pages. Just as users don't like having to spend more than four clicks to get to the content they want, the Google crawler is often unable to crawl great depths of directories if the path is complicated. Crawling can also be influenced by the use of your internal links. on other URLs using in-text deep links. This way, links that point to important content on your homepage will be crawled faster. Using anchor tags to describe the link target gives the bot additional information about what to expect from the link and how to rank the content.
To make the bot crawl your content faster, define your headers logically using h-tags. Here you need to make sure to structure the tags in chronological order. This means using the h1 tag for the main title and h2, h3, etc. for your subtitles. Advertising Continue reading below Many CMSs and web designers often use h tags to format their page header sizes because it's easier. This may disrupt the Google crawler while crawling. You should use CSS to specify font sizes independent of content. 3. Avoid forcing the robot to take detours Orphan pages and 404 errors unnecessarily stress the crawl budget. Each time the Google crawler encounters an error page , it cannot follow any other links and therefore has to go back and start over from a different point.