Let AI agents read your store through robots.txt

Check A1 in the Access dimension, worth 6 points of 100 and measured by software.

What we check

We read your robots.txt the way a crawler does, following the robots.txt standard (RFC 9309), and ask whether each of 8 crawlers may open 4 pages: your homepage, a product, a collection and your return policy. The 8 are Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Applebot and Meta-WebIndexer. Each of them follows robots.txt, and what they read ends up in the answers assistants give shoppers.

The score is the share of the 32 pairs of crawler and page that robots.txt allows. A store with no robots.txt allows everything. A robots.txt that answers with a server error counts as blocking everything, since that's how crawlers treat it.

Crawlers that only collect training data, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, are listed in your report but don't count. Blocking them is a choice about training models, not about being found.

Why it matters

Agents that respect robots.txt skip those pages, so they cannot cite or recommend what is on them.

Each scored crawler feeds a different assistant. OAI-SearchBot fills ChatGPT's search results, Claude-SearchBot and Claude-User serve Claude, and PerplexityBot serves Perplexity. Googlebot's index is what Google's AI Mode reads, and Bingbot's feeds Microsoft Copilot.

How to fix it

Edit robots.txt.liquid in your theme to allow these agents on product, collection and policy paths. Typical effort: 30 minutes.

1. Save a copy of what yourstore.com/robots.txt shows today, so you can compare before and after.
2. Shopify's generated robots.txt lets these crawlers read your public pages, so a block usually comes from a robots.txt.liquid template in your theme, from an app, or from a service in front of your store such as Cloudflare.
3. To edit the template, open Online Store, then Themes, then Edit code in Shopify admin, and look in the Templates folder for robots.txt.liquid. Remove the rules that disallow the crawlers above on product, collection and policy pages.
4. If you want to keep training crawlers out, block them by name rather than blocking every agent. The template below keeps Shopify's own rules and adds one group for GPTBot.

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

User-agent: GPTBot
Disallow: /

Check it yourself

Open yourstore.com/robots.txt and read each group. A Disallow: / under User-agent: *, or under one of the 8 crawlers above, keeps that crawler out of the whole store.