Robots.txt Generator
Free robots.txt generator: create a robots.txt file with disallow rules, a WordPress preset, AI crawler blocking (GPTBot, ClaudeBot and more) and your sitemap.
- 100% free
- No sign-up
- Runs in your browser
Start from a preset
Block AI crawlers (optional)
Stops these bots using your content for AI. It does not change your Google or Bing rankings.
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /?s= Disallow: /search/ Sitemap: https://example.com/sitemap.xml
Webmaster Tools
About the Robots.txt Generator
A robots.txt file tells search engine crawlers which parts of your site they may visit. This generator writes one for you: choose to allow or block all robots, add the folders to keep crawlers out of, and add the address of your sitemap.
Start from a preset for WordPress, an online shop or a simple site, then adjust it. You can add Allow lines for files inside a blocked folder, such as WordPress’s admin-ajax.php, and an optional crawl delay.
You can also block AI crawlers, such as GPTBot, ClaudeBot, Google-Extended and CCBot, from using your content for AI. Blocking them does not change your rankings in Google Search or Bing.
How to use this tool
- 1Choose "Allow all robots" for a normal website.
- 2Pick a preset, or type the paths to block, one per line.
- 3Tick any AI crawlers you want to block.
- 4Enter your sitemap address.
- 5Copy the result into a file called robots.txt and upload it to the root folder of your site.
Best-practice tips
- robots.txt must sit at the root of the domain, so it opens at yourdomain.com/robots.txt.
- Blocking a page in robots.txt does not remove it from Google. To keep a page out of results, use a noindex tag instead and let Google crawl it.
- Never block CSS or JavaScript files: Google needs them to see your pages properly.
- Do not use robots.txt to hide private content. The file is public, and anyone can read it.
robots.txt for WordPress
A good WordPress robots.txt is short. Block the admin area but allow admin-ajax.php, which some themes and plugins need on the public site, and block internal search results, which create endless thin pages:
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/
Sitemap: https://example.com/sitemap.xmlDo not block /wp-content/ or /wp-includes/: they hold the CSS, JavaScript and images Google needs to see your pages. Many SEO plugins let you edit robots.txt from the dashboard.
Should you block AI crawlers?
It is your choice. Blocking AI crawlers stops companies using your pages to train or feed their AI tools, which some publishers want. It may also mean your content is less likely to be quoted in those AI answers.
Either way, blocking them has no effect on normal search: Googlebot and Bingbot are separate crawlers. Google-Extended only controls Google’s AI use of your content, not Google Search. After uploading, test how a crawler reads a page with the Googlebot Simulator.
Answers
Robots.txt Generator FAQs
In the root folder of your website, so it opens at yourdomain.com/robots.txt. It will not work in a subfolder.
Not reliably. A blocked page can still appear if other sites link to it. Use a noindex tag to keep a page out of search results.
No, Google ignores it. Some other search engines, such as Bing, may respect it.
No. They are separate from the crawlers used for search rankings.
Add a line such as Sitemap: https://example.com/sitemap.xml. The generator does it for you.
Yes, with no sign-up.
Related tools
Continue with these related webmaster tools.
Meta Tag Generator
Free meta tag generator: create your title tag, meta description, robots and canonical tags with a live Google search preview and character counter.
View Page Source
Free page source viewer: see the HTML source code of any website with line numbers, a search filter and tag counts. Works on any device, including phones.
XML Sitemap Generator
Free XML sitemap generator: crawl your website (up to 5,000 pages) or paste a URL list and download a valid sitemap.xml. Leaves out redirects, errors and noindex pages.