Robots.txt Generator

Free robots.txt generator: create a robots.txt file with disallow rules, a WordPress preset, AI crawler blocking (GPTBot, ClaudeBot and more) and your sitemap.

  • 100% free
  • No sign-up
  • Runs in your browser

Start from a preset

Block AI crawlers (optional)

Stops these bots using your content for AI. It does not change your Google or Bing rankings.

robots.txt
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

Webmaster Tools

About the Robots.txt Generator

A robots.txt file tells search engine crawlers which parts of your site they may visit. This generator writes one for you: choose to allow or block all robots, add the folders to keep crawlers out of, and add the address of your sitemap.

Start from a preset for WordPress, an online shop or a simple site, then adjust it. You can add Allow lines for files inside a blocked folder, such as WordPress’s admin-ajax.php, and an optional crawl delay.

You can also block AI crawlers, such as GPTBot, ClaudeBot, Google-Extended and CCBot, from using your content for AI. Blocking them does not change your rankings in Google Search or Bing.

How to use this tool

  1. 1Choose "Allow all robots" for a normal website.
  2. 2Pick a preset, or type the paths to block, one per line.
  3. 3Tick any AI crawlers you want to block.
  4. 4Enter your sitemap address.
  5. 5Copy the result into a file called robots.txt and upload it to the root folder of your site.

Best-practice tips

  • robots.txt must sit at the root of the domain, so it opens at yourdomain.com/robots.txt.
  • Blocking a page in robots.txt does not remove it from Google. To keep a page out of results, use a noindex tag instead and let Google crawl it.
  • Never block CSS or JavaScript files: Google needs them to see your pages properly.
  • Do not use robots.txt to hide private content. The file is public, and anyone can read it.

robots.txt for WordPress

A good WordPress robots.txt is short. Block the admin area but allow admin-ajax.php, which some themes and plugins need on the public site, and block internal search results, which create endless thin pages:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

Do not block /wp-content/ or /wp-includes/: they hold the CSS, JavaScript and images Google needs to see your pages. Many SEO plugins let you edit robots.txt from the dashboard.

Should you block AI crawlers?

It is your choice. Blocking AI crawlers stops companies using your pages to train or feed their AI tools, which some publishers want. It may also mean your content is less likely to be quoted in those AI answers.

Either way, blocking them has no effect on normal search: Googlebot and Bingbot are separate crawlers. Google-Extended only controls Google’s AI use of your content, not Google Search. After uploading, test how a crawler reads a page with the Googlebot Simulator.

Answers

Robots.txt Generator FAQs

In the root folder of your website, so it opens at yourdomain.com/robots.txt. It will not work in a subfolder.

Related tools

Continue with these related webmaster tools.