Free tools / Scout
Can Google find all your pages?
Type your website. We find the sitemap, count the pages in it, test a few of them, and read robots.txt for anything blocking Google or AI tools.
- Reading robots.txt
- Finding the sitemap
- Counting pages and testing a sample
What we check
- Whether robots.txt exists, and whether it blocks Google or all crawlers from the whole site.
- Whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot are blocked.
- Where the sitemap is, from robots.txt or the usual addresses, including sitemap indexes from Yoast, Rank Math and Shopify.
- How many pages it lists and when it was last updated.
- Whether a sample of the listed pages actually loads.
Common mistakes
A "Disallow: /" left over from when the site was being built, which hides everything from Google. A sitemap that still lists the old website's pages after a rebuild. And sitemaps full of redirects and missing pages, which tell Google the site is badly kept. All three are quick to fix once you know.
Questions
What is an XML sitemap?
A file that lists the pages on your site you want Google to find, usually at yoursite.com/sitemap.xml. It does not make pages rank on its own, but it helps Google find new and updated pages faster.
What is robots.txt?
A small text file at yoursite.com/robots.txt that tells search engines and other crawlers which parts of the site they may visit. One wrong line, "Disallow: /", can hide a whole site from Google.
Should I block AI crawlers like GPTBot?
It is your choice. Blocking them keeps your content out of AI training and answers. Allowing them gives AI tools like ChatGPT a better chance of recommending your business. Most small businesses benefit from being visible.
Where do I submit my sitemap?
In Google Search Console, under Sitemaps, and in Bing Webmaster Tools. Adding a Sitemap: line to robots.txt also lets every search engine find it.
Want Scout on this every day?
On Lupaan your whole team works in the background and tells you when something needs you.