Sitemaps and getting your pages indexed

A page can only appear in search results if the search engine has indexed it, meaning it has found the page, read it and decided to store it in its enormous database of pages. Many site owners assume that publishing a page is enough. It isn’t always. Some pages are found late, some are found and ignored, and some are accidentally hidden by a setting nobody remembers changing.

Sitemaps and Google Search Console won’t make a weak page rank, but they let you see what search engines know about your site and fix the problems that keep good pages invisible. Here is how they work, using a hypothetical genealogy research society as the example.

Crawling, indexing and ranking in plain terms

Getting into search results happens in three stages, and problems at each stage look different:

1. Crawling. Search engines send automated programs, called crawlers, to discover pages by following links and reading sitemaps.
2. Indexing. The search engine reads what it found and decides whether to store it. It doesn’t store everything; pages that look duplicated, thin or broken may be left out.
3. Ranking. When someone searches, the search engine chooses and orders the indexed pages that best answer the search.

If a page is not indexed, no amount of work on its title or keywords will help. Indexing comes first.

What a sitemap is, and what belongs in it

A sitemap is a file, usually found at an address like yoursite.com/sitemap.xml, that lists the pages you want search engines to know about. It often includes the date each page was last changed. Most membership software and content systems create one automatically.

A good sitemap includes only pages you want in search results:

• Include: public articles, guides, teaser pages, course descriptions, event pages, your home and join pages.
• Leave out: members-only pages, login and account pages, thank-you pages, internal search results, pages marked noindex, addresses that redirect elsewhere and duplicate versions of pages.

Keep the last-changed dates honest, updating them only when a page genuinely changes. Google has said it ignores the priority and change-frequency fields some sitemaps include, so don’t spend time on those.

It also helps to mention your sitemap’s address in your robots.txt file, a small text file at the root of your site that gives crawlers instructions. And remember what a sitemap is: an invitation, not a command. It helps search engines discover pages, but they still decide whether to index them.

Submitting your sitemap in Google Search Console

Google Search Console is Google’s free service for site owners. To submit your sitemap:

1. Add and verify your site in Search Console. Verifying the whole domain, rather than a single version of it, covers every variation of your address.
2. Open the Sitemaps report and enter your sitemap’s address.
3. Submit it, then check back a day or two later. The report should show it was read successfully and how many pages it found.

Other search engines offer similar free tools; submit your sitemap there too.

Checking whether a page is indexed

Two parts of Search Console answer the question “is my page in Google?”

• The URL Inspection tool. Paste in any address from your site and it tells you whether the page is indexed, when it was last crawled and which version Google treats as the main one. For a new or significantly updated page, you can request indexing here. Use that sparingly, for important pages.
• The Page indexing report. This shows how many of your pages are indexed and groups the rest by the reason they are not.

Common indexing reasons, and what they mean

The Page indexing report uses short labels. The ones you will most often see:

• Excluded by noindex tag. The page asks not to be indexed. Correct for members-only and account pages; a problem if it appears on public articles.
• Blocked by robots.txt. Your robots.txt file tells crawlers not to visit. Check that it isn’t blocking public sections by mistake.
• Page with redirect and Not found (404). Usually fine if you moved or deleted the page on purpose. Make sure these addresses aren’t still in your sitemap or internal links.
• Alternate page with proper canonical tag. A duplicate pointing to its main version. Normal and fine; see duplicate content and canonical tags, explained simply.
• Soft 404. The page loads without an error code but looks empty or like an error. On membership sites this often means a members-only page showing “please log in” to the crawler.
• Discovered, currently not indexed. Google knows the address but hasn’t crawled it yet. Often a sign the page has few internal links or the site has many low-value pages competing for attention.
• Crawled, currently not indexed. Google read the page and decided not to store it for now. Usually a sign the page is thin, very similar to others or not clearly useful.

A worked example: the society’s missing guides

The genealogy society has, say, forty public research guides, such as how to find immigration records and how to read old census entries. Search Console shows only twenty-eight of them indexed. Working through the Page indexing report, the society finds:

• Six guides crawled but not indexed. Each was a few short paragraphs, and several overlapped. They are merged into two thorough guides, with redirects from the old addresses.
• Four guides excluded by noindex. They had been created by copying a members-only page template that included the noindex setting. The setting is removed.
• Two guides discovered but not indexed. Nothing on the site linked to them. They are added to the research guides hub page and linked from related articles, following the approach in internal linking: the easiest SEO win you are not using.

The society also discovers that its sitemap lists a hundred and fifty members-only addresses that redirect to the login page. Those are removed so the sitemap lists only pages that belong in search.

Your indexing checklist

1. Verify your whole domain in Google Search Console.
2. Find your sitemap, check that it lists only public pages you want in search, and submit it.
3. Open the Page indexing report and review every reason listed for pages not being indexed.
4. Check that no public pages carry a noindex tag and that robots.txt isn’t blocking public sections.
5. Make sure members-only pages don’t appear in your sitemap.
6. Use the URL Inspection tool on your most important pages, and on each new flagship page after publishing.
7. Review the report monthly, and after any site redesign or platform change.