Use it as a crawler guide, not an index switch.
robots.txt can block crawling, but it does not reliably remove a URL from Google once Google already knows the URL exists.
robots.txt is a plain text file, usually found at /robots.txt. It tells crawlers which paths they are allowed or disallowed to request.
Search engines normally respect it, but it is not a privacy tool and not a reliable way to remove pages from search results.
For pages you do not want indexed, use the right indexation method as well, such as noindex, canonical tags, deletion, redirects, password protection, or Search Console removal depending on the case.
What robots.txt can control 🧭
User-agent, Allow, and Disallowrobots.txt belongs in technical SEO. It works best with a clean XML sitemap, canonical tags, and pages that are allowed to be crawled when they should rank.
One common mistake is blocking a page in robots.txt and adding noindex on the page at the same time. If Google cannot crawl the page, it may never see the noindex instruction.
Robots.txt FAQ 🙋
Back to the dictionary.
What is robots.txt?
robots.txt is a plain text file that gives crawler instructions for a website.
Where is robots.txt located?
It is normally available at the root of a domain, such as https://example.com/robots.txt.
Can robots.txt remove a page from Google?
Not reliably. It can block crawling, but removal from search usually needs noindex, deletion, redirects, password protection, or Search Console removal depending on the case.
Should I add my sitemap to robots.txt?
Yes, it is common to reference the XML sitemap in robots.txt so crawlers can find it quickly.
Can robots.txt block bad bots?
Only bots that choose to respect it. Security and abuse control need server, firewall, or access rules.
Not sure whether robots.txt is helping or blocking the wrong thing?
Send the robots.txt URL and the page you are worried about. We can check whether the issue is crawling, indexing, canonicals, or site structure.