Website data and crawlability
See what AI crawlers can find on your website compared with competitors, and check which AI bots your robots.txt allows or blocks.
Updated September 30, 2026
AI assistants can only recommend what they can read. Website Data shows what AI crawlers can see on your site compared with your competitors, and Crawlability shows which AI bots your robots.txt lets in.
Website Data
Open your project and select the Website Data tab.
Website Overview summarises your crawled site:
| Metric | Meaning |
|---|---|
| Known URLs | Pages discovered on the domain. |
| Total Links | Links found across those pages. |
| Avg. Links | Average links per page. |
| Last Crawled | When the site was last crawled. |
Website Comparison puts your project and competitor domains side by side, with Known URLs, Loaded, Links, Avg. Links, Last Crawled, Top URL and Classification.
Domain Comparison charts crawled coverage and link distribution across domains (as bar, pie or bubble charts), and Website Breakdown shows how links are distributed within a single domain. Use the Competitor selector to focus on a competitor's site.
If a competitor has far more known URLs and links than you, AI models have more of their content to draw from. More useful, well-linked pages give assistants more to cite.
Crawlability
Select the Crawlability tab to see how accessible your site is to AI crawlers.
Bot access rules lists the major AI bots, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended, Amazonbot, Meta-ExternalAgent and Bytespider, with:
- Bot: the crawler's user agent.
- Type: what the bot is used for: Training, Search, or User Query (fetching pages when a user asks a question).
- Access: Allowed, Partial or Blocked.
Bots with their own rules in your robots.txt are listed under Explicitly defined; the rest are under Following global rules. Click a bot to highlight the matching lines in Robots.txt details, which shows your full robots.txt file.
Blocking search or user-query bots can stop an assistant from reading and citing your pages when it answers questions. Review blocked bots with whoever manages your website before changing robots.txt.
If you see "Website data not found", your site hasn't been crawled yet. Rules will appear once it has.
What's next?
Was this page helpful?
Still stuck? Contact support