Free tool

Can Google (and ChatGPT) actually crawl your page?

Q: What does robots.txt do?

It is a plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which parts of the site they may visit. It controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it. To keep a page out of results, use a noindex tag instead, and leave it crawlable so Google can see the tag.

Q: What is the most common robots.txt mistake?

Leaving Disallow: / in place after a site launch. It is often added while a site is being built and forgotten, which blocks the whole site from search engines.

Test any page against its robots.txt for Google, Bing and the main AI crawlers, see exactly which rule decides it, then build a correct file below.

Generator

Build a robots.txt file

Choose what to allow, copy the result and save it as robots.txt in the top folder of your website.

Search engines

Block everything only for a staging or private site.

Folders or pages to keep crawlers out of (one per line)

Start each with a /. Use * as a wildcard. Never block CSS or JavaScript files.

Exceptions to allow inside those folders (optional)

Block AI training crawlers

These collect content to train AI models. Blocking them doesn't affect Google Search or AI search answers.

GPTBot · OpenAI training ClaudeBot · Anthropic training Google-Extended · Gemini training CCBot · Common Crawl Bytespider · ByteDance meta-externalagent · Meta AI training

Sitemap address

robots.txt

Questions

Frequently asked questions

Still have a question?

Ask us directly. You will get a straight answer, even if it is that you don't need us.

Ask Your Question

What does robots.txt do?

It is a plain text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which parts of the site they may visit. It controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it. To keep a page out of results, use a noindex tag instead, and leave it crawlable so Google can see the tag.

How does Google decide when rules conflict?

Google uses the most specific rule: the one with the longest matching path. If an Allow and a Disallow rule are equally long, Allow wins. * matches any characters and $ marks the end of the address. This tester follows the same rules.

Should I block AI crawlers?

It depends on what you want. Blocking training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future model training and does not affect Google Search. Blocking search bots such as OAI-SearchBot, PerplexityBot or Bingbot can stop you appearing in AI answers, which most businesses want to be part of.

What is the most common robots.txt mistake?

Leaving Disallow: / in place after a site launch. It is often added while a site is being built and forgotten, which blocks the whole site from search engines.

Start a conversation

Not sure what should be crawled?

A wrong robots.txt line can hide a whole site from Google. We review crawling, indexing and sitemaps as part of every technical SEO audit.

Request a Technical SEO Audit

Email sumitpandey010@yahoo.com
Phone & WhatsApp +91 62875 39407
Reply time From a real person, within one business day