Does Robots.txt stop a page appearing in Google?

No. Robots.txt restricts crawler access, but a blocked URL may still be discovered through links and shown without a crawled page description.

Where should Robots.txt be placed?

Robots.txt belongs at the root of the relevant website host, such as the main domain. A file placed in a subfolder does not control the whole site.

Can Robots.txt block images or other files?

Yes. Rules can target matching file paths, folders or resource URLs. Blocking a resource can affect how crawlers render or understand pages that depend on it.

Glossary

What Is Robots.txt?

Robots.txt is a plain-text file that gives search engine crawlers instructions about which areas of a website they may request. Website owners place the file at the site’s root address. The file can guide crawling, but it does not remove pages from search results or replace page-level indexing controls such as a noindex tag.

Quick Facts About Robots.txt

Key Takeaways About Robots.txt

Robots.txt controls crawler requests, not guaranteed search visibility.
Rules apply to matching URL paths on the relevant website.
Blocking a page does not equal removing it from search results.

How Robots.txt Works

A hand holds a paper labeled Robots.txt beside a laptop, mouse, notebook, pen, and small plant on a desk

Robots.txt sits at the root of a website and contains rules for named crawler agents. Each rule can allow or disallow requests to matching paths, such as an internal search area or a staging folder. A crawler reads the file before requesting those paths, then follows the instructions it supports.

Related glossary terms: Crawl Budget, Crawl Error, Noindex Tag.

The file manages crawl access, not index status. A disallowed URL may still appear in search results if another page refers to it, although the crawler may lack access to inspect its content. Noindex tags work at page level when a crawler can access the page, while a sitemap provides URLs that a site wants crawled. Technical SEO checks confirm that the file loads correctly, uses valid syntax and does not block important public content.

Hands hold a paper reading “robots.txt” beside a laptop, notebook, and pen on a desk

Expert Note

When we review Robots.txt, we check the live file, its rules and the relationship between blocked paths, important pages and the sitemap. We also test whether the rules affect the correct crawler and confirm that the file does not restrict content that search engines need to access.

Robots.txt in Practice: A Real-World Example

A Geelong service business may use Robots.txt to keep its internal website search URLs from being crawled while leaving service pages open. The file can also point crawlers towards the site’s sitemap, which lists important public URLs.