robots.txt is a small file that gives instructions to search engine crawlers.
It lives at: yourdomain.com/robots.txt
Important: robots.txt is about crawling, not indexing. Blocking a page in robots.txt doesn’t guarantee it won’t appear in search.
What robots.txt is used for 🧭
- Reduce crawling of low-value URLs (filters, parameters, internal search)
- Point to your XML sitemap
- Keep crawlers out of admin areas
This is part of technical SEO. Misuse it and you can block your whole site (yes, it happens).
Robots.txt FAQ 🙋
What is robots.txt?
robots.txt is a file on your domain that gives crawl instructions to search engine bots.
Does robots.txt block indexing?
Not reliably. It blocks crawling. A URL can still appear in results if other pages link to it. Use noindex (when possible) for indexing control.
Where is the robots.txt file located?
At the root of your domain: example.com/robots.txt.
Should I put my sitemap in robots.txt?
Yes, it’s common to include a sitemap line pointing to your XML sitemap.
What’s the biggest robots.txt mistake?
Blocking important pages or accidentally blocking the whole site. Always double-check before deploying changes.
Back to the dictionary.