Robots.txt and noindex: deciding what search engines see

Not everything on a membership site belongs in search results. Checkout pages, account settings, thank-you pages, internal search results and members-only lessons all have good reasons to stay out. Two tools shape what search engines do with your pages: the robots.txt file and the noindex instruction. They sound similar and are often confused, but they do different jobs, and using the wrong one can expose pages you meant to hide or hide pages you meant to show.

This article explains both in plain terms, shows which parts of a typical membership site need which, and covers the combinations that cause the most trouble.

Crawling and indexing are different stages

Search engines work in stages. Crawling is when their software, called a crawler, visits a page and reads it. Indexing is when the search engine stores the page so it can appear in results. The two tools act on different stages:

• Robots.txt controls crawling. It asks crawlers not to visit certain addresses.
• Noindex controls indexing. It lets search engines read a page but tells them to leave it out of results.

That one difference explains nearly every mistake people make with them.

Robots.txt: a request, not a lock

Robots.txt is a small public text file at the root of your site; you can see any site's by adding /robots.txt to its domain. It lists areas crawlers are asked to avoid. A simple file might contain lines like these:

• User-agent: * (the rules that follow apply to all crawlers)
• Disallow: /search (don't crawl internal search results)
• Disallow: /calendar (don't crawl endless calendar pages)
• Sitemap: followed by the full address of your sitemap

Three things to know:

1. It isn't security. Reputable crawlers respect it; others ignore it. And because the file is public, listing private folders in it tells anyone where they are.
2. Blocked pages can still appear in results. If other pages link to a blocked address, a search engine may list it, often with no description, because it knows the page exists but can't read it. Google states plainly that robots.txt is not a way to keep a page out of its results.
3. Its real job is saving crawl effort on large numbers of low-value addresses: internal search results, sorted and filtered views, and calendars that go on forever.

Noindex: in the page, and it must be readable

Noindex is usually a short tag in a page's code; for files such as PDFs, it can be sent as a server instruction instead. Most membership software offers it as a checkbox, often labeled something like “hide from search engines.”

The crucial rule: a search engine has to crawl the page to see the noindex. If robots.txt blocks the page, the noindex is never read, and the page can linger in results. Noindex takes effect after the page's next crawl, and noindexed pages still work normally for visitors and links.

Real privacy comes from the login

Neither tool protects anything. Members-only content stays private because a login is required to see it, and that also keeps it out of search, since crawlers can't log in. Files deserve the same protection; see protecting member-only files and downloads. Test and staging copies of your site should be password-protected, not merely blocked or noindexed.

A worked example: deciding page by page

Rosa runs a calligraphy membership. She lists her page types and decides:

• Home, sales page, public articles and course pages: crawl and index. No action needed.
• Login page: left indexable, because members search for her site's name plus “login.”
• Lessons and practice sheets: protected by the login and kept out of the sitemap.
• Checkout, thank-you and account pages: noindex.
• Internal search results: noindex first, so any already in results drop out, then blocked in robots.txt once they have gone.
• Tag pages with only a handful of articles: noindex, as described in category and tag pages: helpful or harmful for SEO?
• Pages for long-finished sales: removed rather than hidden, as part of pruning; see content pruning.

Her robots.txt ends up with just two blocked paths and a sitemap line. Everything else is handled by the login or by noindex.

Combinations that cause real damage

• Blocking and noindexing the same page. The block stops the noindex from being seen.
• Using robots.txt to remove indexed pages. It stops crawling, not listing. Use noindex or remove the page.
• Rules broader than intended. A rule that blocks /m, meant for one folder, also blocks /members-faq, /mentoring and every other path starting with that letter.
• Blocking the files that build your pages. If style and script files are blocked, search engines may not see pages as visitors do.
• Noindex on a template. One setting on a shared template can hide every page built from it.
• Noindexed pages in your sitemap. A sitemap should list only pages you want in results.

Your crawling and indexing checklist

1. Read your robots.txt file and confirm it blocks only low-value areas, with no overly broad rules.
2. Check the robots.txt report in Google Search Console for errors.
3. Set checkout, thank-you and account pages to noindex.
4. Make sure no page is both blocked and noindexed.
5. Run key public pages through the URL Inspection tool to confirm they can be crawled and indexed.
6. Review the Page indexing report for pages marked as blocked or excluded by noindex that shouldn't be.
7. Protect members-only content and staging sites with a login, never with robots.txt alone.