Robots.txt: Controlling Search Engine Crawlers
The robots.txt file is one of the most important — and most misunderstood — SEO configuration files. This guide explains how it works, how to write it correctly, and common mistakes that can accidentally hide your content from search engines.
Table of Contents
- What Is Robots.txt?
- Robots.txt Syntax and Directives
- Common Robots.txt Patterns
- Testing Your Robots.txt
- Common Mistakes to Avoid
What Is Robots.txt?
Robots.txt is a plain text file located at the root of your website (e.g., https://yoursite.com/robots.txt). It follows the Robots Exclusion Protocol, a standard that tells search engine crawlers which parts of your site they are allowed to visit. When a crawler arrives at your site, it first checks this file before crawling any pages.
The file serves several purposes: protecting private areas of your site from being crawled, managing your crawl budget by preventing crawlers from wasting time on unimportant pages, and specifying the location of your sitemap. However, it is important to understand that robots.txt controls crawling, not indexing — a blocked page can still appear in search results if other sites link to it.
Robots.txt Syntax and Directives
A robots.txt file uses a simple syntax with two main directives. User-agent specifies which crawler the following rules apply to (use * for all crawlers). Disallow tells the crawler which paths it should not access. For example:
User-agent: *
Disallow: /admin/
Disallow: /private/
The Allow directive overrides a Disallow for specific paths — useful when you want to block a directory but allow a subdirectory. The Sitemap directive points crawlers to your XML sitemap. Each set of rules (a "record") starts with a User-agent line and includes one or more Disallow/Allow directives.
Common Robots.txt Patterns
To allow all crawling: User-agent: * / Disallow: (empty Disallow means everything is allowed). To block everything: User-agent: * / Disallow: /. To block a specific directory: Disallow: /private/. To block a specific file: Disallow: /page.html.
You can use the wildcard (*) to match patterns: Disallow: /*.pdf$ blocks all PDF files. The $ symbol marks the end of a URL pattern. To target specific crawlers, replace * with the bot name (e.g., User-agent: Googlebot) to create different rules for different search engines.
Testing Your Robots.txt
Always test your robots.txt file before deploying it. Google Search Console has a robots.txt Tester tool that validates your file and lets you test specific URLs against your rules. You can also manually check by visiting yoursite.com/robots.txt in your browser.
After making changes, monitor Google Search Console's Coverage report to ensure important pages are still being indexed and unimportant pages are being properly excluded. Remember that changes to robots.txt may take time to be processed by search engines.
Common Mistakes to Avoid
The most dangerous mistake is accidentally blocking your entire site with Disallow: /. This has happened to major websites, causing them to disappear from search results. Another common error is blocking CSS and JavaScript files, which prevents Google from properly rendering your pages and understanding your content.
Do not use robots.txt to hide sensitive information — it is publicly accessible to anyone. Use proper authentication for private content. Avoid listing every single page individually; use directory-level rules instead. And remember that not all crawlers respect robots.txt — malicious bots will ignore it entirely.
Try it now: Open the Robots.txt Generator →
Frequently Asked Questions
What is robots.txt?
Robots.txt is a text file placed in the root directory of a website that instructs search engine crawlers which pages or sections they can or cannot access. It uses a standard protocol called the Robots Exclusion Protocol to communicate crawling rules.
Does robots.txt prevent indexing?
No. Robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. To prevent indexing, use the noindex meta tag or X-Robots-Tag HTTP header instead.
Where should robots.txt be placed?
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt). It must be named exactly 'robots.txt' (lowercase) and accessible at that exact URL for crawlers to read it.