robots.txt vs noindex: blocking crawling versus keeping a page out of the index

robots.txt controls crawling: it tells a crawler which URLs not to fetch. noindex controls the index: it tells a search engine not to show a page in its results. Both of them "block" something, so people mix them up all the time, and the most expensive version of the mistake goes like this. Someone disallows a page in robots.txt to get it out of search, and the page stays in the results, now with no description. This article covers how each mechanism behaves in Yandex and Google, what to do with parameterised URLs, when you need X-Robots-Tag, and how to confirm your rules do what you meant.

Crawling and indexing are separate decisions

A page reaches search results in three steps. The crawler learns the URL exists (from a sitemap, a link or a ping), it fetches the page, and then the search engine decides whether to add the page to its index. That path, and how to check each step, is covered in our guide to checking whether your site is indexed.

robots.txt acts on the second step. It forbids fetching a URL; it does not stop the engine from knowing the URL exists. Google's documentation says plainly that robots.txt is not a mechanism for keeping a web page out of Google, and that its job is to limit how many requests a crawler makes to your site (developers.google.com, checked 15 October 2026).

noindex acts on the third step. The crawler fetches the page, reads the directive, and leaves the page out of the index, or removes it if it was already there. That is why noindex only works on a page the crawler is allowed to fetch. A crawler barred from downloading the page never sees the directive.

If you want crawlers to stop spending requests on useless URLs, you want robots.txt. If you want a page out of the results, you want noindex, a 404 or 410 status, or a password. Sometimes you want both, and then the order matters: deal with the index first and with crawling second.

How robots.txt works

The file sits at the root of the host (https://example.com/robots.txt) and is made of groups. Each group opens with a User-agent line naming the crawler, followed by Disallow and Allow rules with path prefixes. A separate Sitemap line gives the full URL of your sitemap.

The syntax details people get wrong most often:

• Paths are case-sensitive. Disallow: /Catalog/ does not block /catalog/. Google's specification says so explicitly.
• An asterisk matches any sequence of characters, including none. Every rule has an implied asterisk at the end, so Disallow: /search blocks both /search and /search-results/. A $ cancels that: Disallow: /example$ blocks only /example itself.
• Line order in the file does not matter. Yandex sorts the rules in a group by prefix length and applies the longest one that matches, and when two matching rules are the same length, Allow wins (Yandex help, checked 15 October 2026). Google also picks the most specific rule, and when rules conflict it uses the least restrictive one.
• An empty Disallow: is the same as Allow: /: everything is allowed.

There is a size limit. Google and Yandex both give 500 KB, and Google adds that anything after that point is ignored. Google usually caches robots.txt for up to 24 hours, so an edit does not take effect the moment you upload it.

It is worth knowing what happens when the file itself fails. Under Google's specification, a 4xx response (other than 429) counts as "no file", and Google crawls without restrictions. A 5xx is worse: Google stops crawling the site for the first 12 hours, then falls back on the last good copy for up to 30 days. Yandex puts it more simply: if the file does not return 200 or does not meet its requirements, the site is treated as open for indexing. So a broken robots.txt can either open everything or halt Google's crawling for half a day.

Why Disallow does not remove a page from search

Both search engines say this in so many words. Google: a page blocked by robots.txt can still appear in search results if other pages link to it, but the result will have no description. Yandex, in its robots.txt help: pages restricted in robots.txt can take part in Yandex search, and removing them from search takes a noindex directive in the HTML or in an HTTP header (both pages checked 15 October 2026).

The engine knows the URL from internal and external links and knows the anchor text of those links, but it has never seen the content. With enough signals, the URL goes into the results as a bare link, with a title pieced together from anchors and an empty snippet or a note that no description is available. Search Console files such pages under "Indexed, though blocked by robots.txt".

The reverse case is nastier. A page already carries noindex, and someone adds it to robots.txt to be extra safe. Now the crawler cannot fetch it, never sees the directive, and the page that was supposed to drop out of the index stays in. Google spells this out: if a page is disallowed from crawling through robots.txt, its indexing rules will not be found and will be ignored. Yandex's help on meta tags says the same: if the page is blocked in robots.txt, the meta tag or header directive has no effect.

So Disallow plus noindex on the same page is almost always a mistake. The crawl block wins and the noindex goes unseen.

noindex and X-Robots-Tag

You can send noindex in two ways. One is a meta tag in the page's <head>: <meta name="robots" content="noindex">. The other is an HTTP response header, X-Robots-Tag: noindex. For HTML pages they are equivalent. For files that have no HTML, such as PDFs, images and documents, the header is the only option. Google's meta tag documentation gives an Apache example that applies Header set X-Robots-Tag "noindex, nofollow" to every file ending in .pdf. In nginx you do the same with add_header inside the relevant location.

The header is handy for HTML too when you cannot touch the templates: one line of server config can cover a whole section instead of a hundred page edits. The downside is that the header is invisible in the page source, and a year later nobody remembers why the section vanished from search. Write such rules down next to the config.

Besides noindex, both engines understand nofollow (do not follow links on the page) and none (noindex and nofollow together). Google's list is longer: nosnippet, max-snippet, noimageindex, and unavailable_after with a date after which the page should leave the results. In the X-Robots-Tag header, Yandex's help lists support only for the restrictive directives: noindex, nofollow, none and noarchive.

The two engines resolve conflicts in opposite directions. If a page carries contradictory directives, say one robots tag with all and another with noindex, Yandex picks the permissive one: its help says permissive directives take priority. Google does the reverse: when robots rules conflict, the more restrictive rule applies. The same page with two tags ends up in Yandex's index and out of Google's. The simple fix is never to have two robots tags on one page; if your CMS and an SEO plugin each print their own, keep one.

noindex is not instant either. Google reminds you that a page leaves the results only after it is crawled again, and for a rarely visited page that can take months. A recrawl request in Search Console or in Yandex Webmaster speeds it up.

Clean-param: how Yandex merges parameterised URLs

On many sites the same page is reachable at dozens of URLs: with utm_* tags, a session ID, an affiliate parameter. Blocking those with Disallow is clumsy, and noindex is the wrong tool because it is the same page and you want it indexed. Yandex has its own directive for this, Clean-param.

The syntax from Yandex's help (checked 15 October 2026) is Clean-param: p0[&p1&p2&..&pn] [path]. The first field lists the parameters the crawler should ignore, separated by ampersands; the optional second field is the path prefix the rule applies to. The help's example, Clean-param: ref /some_dir/get_book.pl, collapses URLs like get_book.pl?ref=site_1&book_id=123 into a single get_book.pl?book_id=123.

For a typical site with campaign tags the rule looks like this: Clean-param: utm_source&utm_medium&utm_campaign&utm_content&utm_term. The help sets three limits: a rule may be at most 500 characters, it is case-sensitive, and the directive is cross-sectional, so it can go anywhere in the file.

Only Yandex reads Clean-param. Google skips the line, so for Google you consolidate the same duplicates with a rel="canonical" on the page pointing at the parameter-free URL. In practice you need both: Clean-param saves Yandex's crawling, and canonical pulls Google's signals onto one URL.

While you are in the file, two directives still turn up in old robots.txt files. Yandex has ignored Crawl-delay since 22 February 2018 and offers a crawl rate setting in Webmaster instead. Google does not support crawl-delay either. Host is not mentioned at all in Yandex's current robots.txt help. Both lines can go.

What to use for which pages

The choice comes down to two questions: should crawlers spend time on these URLs, and should the page ever show up in search?

Which filters earn a landing page of their own is a keyword question, and robots.txt cannot answer it: if a combination has real search volume, it belongs in the site structure. How to tell is covered in our guide to building a semantic core.

Common mistakes

Nearly every robots.txt and noindex problem we find in audits is one of a handful of repeat cases.

• Disallow used for removal. Someone wants a page out of search and blocks it in robots.txt. It stays in the results with no description, and if it had a noindex before, the crawler can no longer see it.
• A block carried over from staging. The staging copy has Disallow: / or a noindex in the template, and the file ships with the release. A week later traffic starts to fall. Checking robots.txt and the homepage headers belongs on every release checklist.
• Blocked CSS and JavaScript. An old Disallow: /assets/ or /wp-includes/ stops the engine from seeing the page the way a person does. More on this in our technical audit checklist.
• Case and short prefixes. Disallow: /Admin does not block /admin, and Disallow: /blog also blocks /blog-old/ and /blogger/.
• noindex on the canonical page. Duplicates point to the main page with canonical, and the main page carries a stray noindex. The engine gets contradictory signals and the whole group can drop out.
• Two robots tags from different plugins. Yandex goes with the permissive one, Google with the restrictive one, and the two engines end up treating the page differently.
• noindexed or disallowed URLs in the sitemap. A sitemap should list only URLs you want indexed that return 200.

How to check: Yandex Webmaster and Search Console

Rules are easiest to check before they ship, and both engines give you tools for it.

In Yandex Webmaster the tool is called robots.txt analysis (Анализ robots.txt), under Tools. According to Yandex's help (checked 15 October 2026), it loads the site's current file, flags syntax errors, and lets you paste your own version of the rules and test it before publishing. In the page access block you can list URLs and see which ones are open for indexing and which are not. Webmaster keeps up to 100 saved versions of the file, so you can find out when a bad rule appeared.

For a single page, Webmaster has a URL status check that shows whether Yandex knows the URL, when it last crawled it, and whether the page is in search. If the exclusion reason says robots.txt or noindex, you know which rule fired.

Google Search Console has a robots.txt report in its settings. Per Google's help, it shows which robots.txt files Google found for the top 20 hosts on your site, when each was last checked, its size, and any problems found. You can request a recrawl of the file from there too, though Google says to do that only in urgent cases.

To check whether a specific URL is blocked, Search Console's URL Inspection tool is the better fit. It shows whether crawling is allowed, whether indexing is allowed, and, if not, what is in the way: robots.txt, or noindex in a meta tag or header. The live test fetches the page now, so it sees fresh changes.

The quickest way to check an X-Robots-Tag header is the command line: curl -sI https://example.com/file.pdf prints the response headers without the body. If the header is not there, the server rule did not apply, whatever the config file says.

When a page has to come out of search

This order works in both engines:

1. Make sure the URL is not disallowed in robots.txt. If it is, open it up for now.
2. Add noindex (meta tag or header), or return 404 or 410 if the page is gone for good. For confidential data there is only one right answer: take it off the site or put it behind a password.
3. Ask the crawler to come back: a recrawl in Yandex Webmaster, an indexing request in Search Console.
4. For urgent cases Google has the Removals tool. It hides a URL from results for about six months (developers.google.com, checked 15 October 2026) but does not drop it from the index, so use that time to add noindex or remove the page. Yandex Webmaster has a similar section for removing pages from search.
5. Wait until the page has dropped out of both engines, and only then, if you want to save crawling, disallow the section in robots.txt.

People often do it the other way round: disallow first, add noindex later, and the page sits in the results without a description for weeks. If the drop in indexed pages shows up in a client report, explain the cause there too, or a smaller index count looks like an accident. For the other places where the two engines need separate handling, see our piece on Yandex and Google.

Frequently asked questions

Can I keep a page out of the index with robots.txt?

No. robots.txt blocks crawling, not indexing. Google and Yandex both say a page disallowed in robots.txt can still appear in search if other pages link to it. Use noindex or a 404/410 status for the index.

Should I use noindex and Disallow together?

Usually not. While the page is disallowed in robots.txt, the crawler cannot see the noindex. Add noindex first, wait for the page to drop out, and only then disallow it if you need to.

Does a noindex line inside robots.txt work?

No. Neither Google nor Yandex lists it among supported robots.txt rules. noindex goes in a meta tag on the page or in an X-Robots-Tag header.

Does Google understand Clean-param?

No, it is a Yandex directive. For Google, consolidate parameterised duplicates with a rel="canonical" pointing at the URL without parameters.

How quickly does a noindexed page disappear from results?

After its next crawl. For frequently crawled pages that means days; for rarely visited ones it can take months. A recrawl request in Yandex Webmaster or an indexing request in Search Console speeds it up.

What happens if robots.txt returns a server error?

Under Google's specification, a 5xx stops crawling of the site for the first 12 hours, after which Google uses the last saved copy of the file. Yandex treats a site whose robots.txt is unavailable as open for indexing.