Skip to content
DA PA Checker

Free tool

Robots.txt Tester and Generator

Check if Google can crawl a URL, read the rules, or build a new robots.txt.

The robots.txt tester reads a site’s robots.txt file and tells you if Google, Bing and other bots may crawl a given path. The generator below builds a clean file for your own site.

One wrong line in robots.txt can block a whole site from search, so it is worth checking.

How to use it

  1. Enter a domain and, if you like, a path such as /blog/.
  2. Press the orange button.
  3. Read whether each bot is allowed or blocked, and see the full file.

Rules in short

  • Disallow blocks crawling of a path.
  • Allow makes an exception inside a blocked path.
  • Sitemap tells bots where your sitemap lives.

Tip

Robots.txt blocks crawling, not indexing. To keep a page out of Google, use a noindex tag instead.

Questions

Should I block /wp-admin/?

Yes, but allow /wp-admin/admin-ajax.php, as the generator does.

Can I block AI bots?

Yes. Tick the option in the generator to block common AI training bots.

Is robots.txt private?

No. Anyone can read it, so do not list secret paths in it.

More free tools

DA PA CheckerDomain and Page Authority with DC Score.Bulk DA PA CheckerCheck up to 200 domains at once.Compare DomainsYour site against four competitors.Page Authority CheckerCheck the Moz Page Authority of any page, plus DA, spam score and DC Score.Spam Score CheckerFind out how spammy a website looks to search engines before you link to it or buy it.Domain Age CheckerSee when a domain was registered, when it expires and when it was first archived.WHOIS LookupRegistrar, registration and expiry dates and name servers of any domain.DNS LookupA, MX and NS records of any domain in one place.SSL CheckerCheck if a site has a valid SSL certificate, who issued it and when it expires.Redirect CheckerFollow every redirect of a URL and see each status code, 301, 302 and more.Website Traffic CheckerEstimate the monthly visits of any website from its global traffic rank.SEO Audit ToolScan any page for 20 SEO checks: titles, headings, speed, indexing, schema and more, with clear fixes.Guest Post Site CheckerIs a site worth a backlink? Authority, spam, traffic, outbound links and age in one verdict.Outbound Links CheckerSee every external link on a page, with dofollow or nofollow and the DC Score of each linked site.Sitemap CheckerFind a site's sitemap, count its URLs and test that the pages load.Keyword Ideas GeneratorHundreds of long-tail keyword and question ideas from real search suggestions.Domain Name GeneratorBrandable domain ideas from one keyword, with live availability.Website Uptime MonitorWe check your site every 5 minutes and email you the moment it goes down.Page Speed CheckerGoogle PageSpeed scores for mobile and desktop, Core Web Vitals and what to fix first.Broken Link CheckerTest every link on a page and find the ones that return errors.SERP Preview ToolSee how a page looks in Google and on social media, and edit the title and description live.Schema Markup GeneratorFill a short form and copy ready JSON-LD for FAQ, Article, Organization or Local Business.Word CounterWords, characters, sentences, reading time and keyword density as you type.UTM Link BuilderBuild tracking links for Google Analytics in seconds.Redirect GeneratorCreate 301 redirects, force HTTPS and www rules for Apache and LiteSpeed.Website Worth CalculatorAn estimate of what a website could sell for, from its traffic, age and authority.