A sitemap remains one of the simplest files to produce on a website. Its function seems obvious: to list URLs so that search engines can find them. In practice, most sitemaps generated automatically by CMS extensions contain unnecessary URLs, incorrect modification dates, and no sorting logic.
The result: a file that Googlebot reads without deriving much from it, and an HTML sitemap page that visitors ignore. Optimizing a sitemap is first about understanding what robots and humans expect from this file, and then deciding what deserves to be included.
Lastmod, obsolete ping, and real signals sent to search engines
The lastmod tag in the XML sitemap is often misused. Google specifies that this value only helps plan recrawls if it reflects a real change in the main content, structured data, or internal links. A CMS that regenerates the sitemap and updates all dates with each build sends a mixed signal: the robot can no longer distinguish a genuinely updated page from an unchanged one.
This technical point has direct consequences on indirect navigation. A sitemap with reliable dates allows engines to prioritize crawling fresh pages. On an editorial or e-commerce site, this means that new content appears faster in the results, reducing the time between publication and the first organic visit.
Another signal to check: the notification method. Google removed the old ping endpoint for sitemaps in June 2023. Plugins that sent a request to this URL no longer notify anything. The two methods that still work are declaration in robots.txt and submission via Google Search Console. A site that hasn’t updated its configuration since that date may believe its new content is being reported when it is not.
For sites also targeting Bing, the IndexNow protocol complements the sitemap by signaling added, modified, or deleted URLs as they occur. This mechanism does not replace the sitemap (which remains an inventory), but it speeds up the acknowledgment of changes by participating engines. When one wishes to view the sitemap page of Agri Systems, one observes an example of structure where each listed URL corresponds to content that is actually accessible and indexable.
XML sitemap and HTML sitemap: two files, two distinct audiences
The confusion between XML sitemap and HTML sitemap page persists because both share the same name. Their recipients have nothing in common.
The XML sitemap is aimed at crawling robots. It lists canonical URLs with their metadata (modification date, possibly change frequency, although Google now ignores the priority and changefreq tags). Its format is not intended to be read by a human.
The HTML sitemap page is aimed at visitors. It functions as a visual site map, organized by categories or themes. Its usefulness depends on the size of the site and the quality of the main navigation menu.
On a site with fewer than fifty pages and a well-structured menu, this HTML page adds almost nothing. On a large site with content buried at multiple levels of depth, it can reduce the number of clicks needed to reach a specific page.
What the HTML sitemap changes for internal linking
An HTML sitemap creates a direct internal link to each URL it lists. This reduces the crawl depth of certain pages, which can facilitate their discovery by engines. But an HTML sitemap does not compensate for a failing internal linking structure. If a page is only accessible via the sitemap and not through any contextual links in the body of articles, it remains weakly connected to the rest of the architecture.
The HTML sitemap works better as a safety net than as a linking strategy. It catches forgotten pages, not those that deserve to be highlighted.
Filtering URLs in the sitemap: what deserves to be included and what to exclude
Most automatically generated sitemaps include all public URLs of the site. This default behavior poses a problem: it drowns strategic pages in a list of tag pages, internal search results, pagination, or duplicate content.
An effective sitemap applies a strict filter. The URLs it contains must meet three simultaneous conditions:
- The page returns an HTTP 200 code, without redirection or error. A URL redirected with a 301 in the sitemap wastes crawl budget and sends a contradictory signal to the robot.
- The page has a meta robots tag allowing indexing. Including a noindex URL in the sitemap is like asking the engine to visit a page to tell it not to keep it.
- The page contains distinct and useful content for a visitor. Faceted filter pages, date archives without proper editorial content, and internal search results pages generally do not belong in the sitemap.
Each URL in the sitemap must be a page you want to see appear in search results. If not, it clutters the file.
Specialized sitemaps by content type: images, videos, news
The XML sitemap format supports extensions dedicated to certain types of media. An image sitemap allows for referencing visuals that would not be discovered by standard crawling (images loaded via JavaScript, for example). A video sitemap provides metadata such as duration, description, and thumbnail, which facilitates appearance in Google’s rich results.
The news sitemap, on the other hand, concerns news sites eligible for Google News. It signals articles published in the last 48 hours with their publication date and title.
These specialized sitemaps are only relevant for sites that produce these types of content in significant volume. A ten-page showcase site does not need a video sitemap. Conversely, an e-commerce site with thousands of illustrated product listings benefits from declaring a separate image sitemap to increase its visibility in Google Images.
Known limitations of these extensions
Google does not guarantee the indexing of media declared in a specialized sitemap. The file facilitates discovery, not selection. Content quality criteria, the relevance of the host page, and popularity signals remain decisive. A perfectly structured video sitemap is not enough if the page hosting the video has no incoming links or traffic.
Maintaining these files also represents a technical burden. An image sitemap that is out of sync with the product catalog (deleted images, broken URLs) degrades the overall reliability of the sitemap in the eyes of the robot. It is better to have a single clean XML sitemap than a set of poorly maintained specialized files.
