Devenia Learn / Dictionary
Use robots.txt as crawler guidance
robots.txt can limit crawling, but it cannot reliably remove a known URL from search results. Learn how to use it with the rest of your technical SEO controls.
Start with email. Share the page or your question, and we will answer clearly.
Choose the control that matches your purpose.
Use robots.txt to guide crawlers away from selected paths. For a public page that should stay out of Google, use noindex and let Google crawl the page to read it.
For private client documents, require a login or other access protection. A robots.txt rule does not stop someone opening the address. Google explains these limits.
Read the rule before changing it
These examples explain individual rules. Check the complete file before applying any change.
Which paths does it cover?
In a group for a particular crawler, Disallow: /drafts/ asks it not to crawl paths beginning with /drafts/. A rule containing only Disallow: / covers every path.
Where is the sitemap?
A Sitemap: line gives the full address of a sitemap. Open that address and check that it is your actual sitemap before adding it.
Which crawler receives the rule?
User-agent: Googlebot addresses Googlebot. User-agent: * is the general group. More specific groups can change which rules apply; read the whole file. See Google’s syntax guide.
Common questions about robots.txt
Back to the dictionary.
What should I check before changing robots.txt?
Save the current file. Write down a URL you want blocked and one that must stay crawlable. Check which rule group applies to your target crawler, then check both URLs after the change.
Where is robots.txt located?
It belongs at the site root, such as example.com/robots.txt. A copy in /folder/robots.txt does not control the site. Subdomains need their own file.
Can robots.txt remove a page from Google?
Do not use it for this purpose. If Google cannot crawl a page, it cannot read that page’s noindex instruction. A known URL may still appear in results. Use Google’s noindex guidance for public pages you want excluded.
Should I list my sitemap in robots.txt?
It is one way to help crawlers discover the sitemap. Use its full address. Listing a sitemap does not override a Disallow rule or guarantee that its pages will be indexed.
Can robots.txt block malicious bots?
No. A bot can ignore these instructions. Protect private content with access controls and handle unwanted traffic through your hosting or security controls.
Unsure whether robots.txt is blocking the wrong thing?
Send the robots.txt address, an affected page, and what you want to happen: allow crawling, keep the page out of search, or restrict access. We can check whether the current rule fits that purpose.