Robots.txt Generator

Disallow does not keep a page out of search results

This is the misunderstanding that matters, because the fix people reach for makes it worse. Disallow controls crawling, not indexing. It tells a crawler not to fetch a URL. It says nothing about whether that URL may be listed, and search engines routinely list disallowed URLs they have learned about from links elsewhere — shown without a description, precisely because the crawler was never allowed to read the page.

Keeping something out of an index needs a noindex directive on the page itself. And a noindex can only be obeyed if the crawler is allowed to fetch the page and see it. So this combination, which looks like belt and braces, is self-defeating:

Disallow: /private/          ← in robots.txt
<meta name="robots" content="noindex">   ← on the page

The Disallow prevents the crawler from ever reading the noindex. Pick one: allow crawling and use noindex, or accept that a disallowed URL may appear as a bare link.

For anything genuinely private, neither is the answer. robots.txt is a public file at a predictable address that lists exactly the paths you would rather people did not visit, and honouring it is entirely voluntary. If a URL must not be reached, it needs authentication, not a polite request.

And the longest matching rule wins, not the first

Rules are not applied in order. The most specific match — the longest pattern — decides, and where two patterns are the same length the less restrictive one wins, which means Allow. Reversing the order of the lines never changes the outcome.

That produces results which look wrong until you count characters. Given these rules:

Disallow: /admin/
Allow: /admin/public/
Disallow: /*.pdf$
Allow: /reports/

The two bold rows are the surprise. /a.pdf is blocked, but /reports/x.pdf is allowed — because /reports/ is nine characters and /*.pdf$ is seven, so the Allow is the longer match and wins. A broad Allow can override a narrow Disallow simply by being a longer string.

The practical consequence: writing rules in an order that reads correctly from top to bottom is no guarantee they behave that way, and the file gives no hint when they do not. Every result in that table is computed by a matcher checked against the documented precedence rules on each build, including the cases where length contradicts reading order.

The pattern that does work reliably is the nested one — block a directory, then permit one path inside it — and it works because the inner path is a longer string, not because it comes second. Google Search Console has a robots.txt tester that will tell you which line decided a given URL, which is faster than reasoning about it.