Devenia Learn / Dictionary
robots.txt
Trying to understand robots.txt, or how it affects your website? Ask us directly and we will point you to the practical next step.
Email first. Send the page or question in writing, and we will reply with a concrete answer.
Use the topic as a starting point.
If this page connects to a real website, ranking, conversion, or technical issue, we can help translate it into action.
robots.txt is a small file that gives instructions to search engine crawlers.
It lives at: yourdomain.com/robots.txt
Important: robots.txt is about crawling, not indexing. Blocking a page in robots.txt doesn’t guarantee it won’t appear in search.
What robots.txt is used for 🧭
- Reduce crawling of low-value URLs (filters, parameters, internal search)
- Point to your XML sitemap
- Keep crawlers out of admin areas
This is part of technical SEO. Misuse it and you can block your whole site (yes, it happens).
Robots.txt FAQ 🙋
What is robots.txt?
robots.txt is a file on your domain that gives crawl instructions to search engine bots.
Does robots.txt block indexing?
Not reliably. It blocks crawling. A URL can still appear in results if other pages link to it. Use noindex (when possible) for indexing control.
Where is the robots.txt file located?
At the root of your domain: example.com/robots.txt.
Should I put my sitemap in robots.txt?
Yes, it’s common to include a sitemap line pointing to your XML sitemap.
What’s the biggest robots.txt mistake?
Blocking important pages or accidentally blocking the whole site. Always double-check before deploying changes.
Back to the dictionary.
Have a practical question about robots.txt?
Send it in context. If it touches SEO, website structure, tracking, content, or conversion, we can help you decide what to do next.