Your site may contain dozens, even hundreds of pages, and yet some remain invisible in search results. The sitemap is the file that signals each of these pages to search engines. Without it, recent content or deep pages in the hierarchy may never be crawled by indexing bots.
What Google does not crawl without explicit indication
Google’s bots do not crawl a site page by page like a human visitor. They follow internal links, and when content is not linked to any other page (or linked from a page that itself is rarely visited), it becomes orphaned in the eyes of the engine.
The XML sitemap solves this problem by providing a complete list of the URLs you want to see indexed. Each entry can include a lastmod tag, which indicates the date of last modification. This is actually the only metadata tag still considered by Google: the priority and changefreq tags have been ignored for several years.
Specifically, if you publish an article today, the lastmod value signals to the bot that this page deserves a new visit. Without this indication, the bot will return according to its own schedule, sometimes weeks later. A good example of practical implementation can be found on the sitemap page of Boost 4 Business, which illustrates how to structure this file for a professional site.
Submitting the XML sitemap: the only two valid methods
You may have read tutorials advising to send a “ping” to Google to signal a new sitemap. This method is outdated. Google disabled the ping endpoint in June 2023: calling the URL /ping?sitemap= no longer triggers anything.
Today, two methods work to submit a sitemap to Google:
- Manual submission via Google Search Console, in the “Sitemaps” section, where you paste the URL of your XML file and validate.
- Declaration in the robots.txt file with the Sitemap: directive followed by the full URL of the file. Bots read this file on every visit and automatically detect the sitemap.
For sites that publish very frequently and want to automate the process, the Search Console API (sitemaps resource) replaces the old ping. This approach requires developer access but ensures that every update to the sitemap is properly accounted for.
Sitemap and orphan pages: the real gain for large sites
On a small site of five pages with a clear navigation menu, the sitemap provides only marginal benefit. Google will find your pages by following internal links. The gain becomes significant in three specific situations.
First, sites with a high volume of content. When a blog exceeds several hundred articles, older posts gradually slip out of reach of the bots. The sitemap keeps these pages within the crawl scope, even if no recent internal links point to them.
Next, e-commerce sites. Product listings are often generated dynamically and depend on category filters. A product accessible only through three levels of filtering is unlikely to be crawled without a sitemap.
Finally, new sites without backlinks. A new site does not yet benefit from incoming links that attract bots. The sitemap submitted via Search Console accelerates the first wave of indexing by directly signaling the pages to be crawled.
Pages to include and pages to exclude
Not all URLs are meant to be included in the sitemap. Only include the pages you want to appear in search results. Login pages, paginated duplicates, internal search results, or noindex pages do not belong there.
A clean sitemap makes the bot’s job easier and avoids wasting crawl budget on URLs without SEO value. If your file contains URLs returning 404 errors or redirects, Google eventually considers the sitemap unreliable.
Sitemap for images and videos: an underutilized lever
The classic XML sitemap lists web pages. But Google also accepts sitemaps dedicated to images and videos. These specific files allow you to signal multimedia content that would otherwise be difficult for a bot to discover.
An image sitemap indicates the URL of each visual, making it easier for them to appear in Google Images. For a showcase site or portfolio, this additional traffic can represent a significant portion of visits.
The video sitemap works on the same principle: it provides the title, description, and thumbnail of each video hosted on your site. Without this declaration, Google may struggle to associate a video with the page hosting it, especially if the video player is loaded via JavaScript.
One file or multiple files
There is no obligation to group everything into a single sitemap. The common practice is to create an index file (sitemap_index.xml) that points to several specialized sitemaps: one for pages, one for articles, one for images. This segmentation makes diagnosis easier when a type of content poses an indexing problem.
Common errors that nullify the effect of the sitemap
Generating a sitemap is not enough. Several common errors render its effectiveness null.
- Leaving URLs with 404 errors or 301 redirects in the file. The bot wastes resources and loses trust in the reliability of the sitemap.
- Never updating the lastmod tag, or worse, updating it artificially without any real content change. Google detects this behavior and eventually ignores this information.
- Including URLs blocked by the robots.txt file. The bot cannot access these pages, creating a contradiction between the sitemap and the crawling directives.
- Exceeding the size limit. An XML sitemap file must not exceed 50 MB uncompressed or contain more than 50,000 URLs. Beyond that, it needs to be segmented.
A sitemap consistent with the rest of the technical setup (robots.txt, noindex tags, canonicals) reinforces your site’s credibility with search engines. Conversely, contradictory signals slow down indexing and may exclude pages you wanted to make visible.
The sitemap does not replace good internal linking or quality content. It acts as a technical safety net that ensures every published page has a chance to be discovered. On active sites, regularly checking its status in Search Console remains the best way to ensure nothing slips through the cracks.
