Devenia Learn / Dictionary
Use robots.txt as crawler guidance
robots.txt can limit crawling, but it cannot reliably remove a known URL from search results. Learn how to use it with the rest of your technical SEO controls.
Start with email. Share the page or your question, and we will answer clearly.
Use it as crawler guidance, not an indexing switch.
robots.txt is a plain-text file, usually available at /robots.txt. It tells crawlers which paths they may request and which paths they should avoid.
It is not a privacy tool and is not a reliable way to remove pages from search results. For pages that should not appear, choose the control that fits the situation: noindex, deletion, a redirect, access protection, or an appropriate removal request.
What robots.txt can control
Use these directives to guide crawling and help crawlers find the sitemap.
Selected paths
XML sitemap discovery
Crawler-specific rules
Common questions about robots.txt
Back to the dictionary.
What is robots.txt?
robots.txt is a plain-text file that gives crawlers instructions for a website.
Where is robots.txt located?
It is usually at the root of a domain, for example https://example.com/robots.txt.
Can robots.txt remove a page from Google?
Not reliably. It can block crawling, but removal usually requires noindex, deletion, redirects, access protection, or an appropriate Search Console process.
Should I list my sitemap in robots.txt?
Yes. Listing the XML sitemap helps crawlers find it quickly.
Can robots.txt block malicious bots?
Only bots that choose to follow the file. Security and abuse protection require server, firewall, or access controls.
Unsure whether robots.txt is blocking the wrong thing?
Send the robots.txt URL and the page you are concerned about. We can check crawling, indexing, canonicals, and site structure.