Let honest crawlers past your bot protection

Bot protection is meant to stop scrapers and fraud. Set too tight, it also turns away the crawlers and assistants that announce exactly who they are.

Check A2 in the Access dimension, worth 5 points of 100 and measured by software.

What we check

GlassAisleBot fetches the pages your robots.txt lets it read, with a user agent that names it and links to our bot page. A response counts as blocked when it's a 403, 429 or 503 that carries the signature of a bot wall, such as a Cloudflare challenge, Akamai, DataDome, HUMAN, Imperva or Kasada, or when a page that answers 200 is really a challenge page. The score is the share of requests that weren't blocked.

We identify ourselves on purpose. A crawler that says who it is and follows robots.txt is the kind of visitor you want to let in, so a wall that stops us is likely to stop the assistants' crawlers too.

Why it matters

Agents that identify themselves meet the same wall and may skip your store.

A challenge page looks like an error to software. The agent can't solve it, so it moves on to a store it can read, and a shopper asking for a recommendation never hears about yours.

How to fix it

Allow verified bots and agents in your bot protection settings instead of challenging all automated traffic. Typical effort: 1 hour.

  1. Your report names the signature each blocked request matched, which tells you which service did the blocking.
  2. If your domain runs through your own Cloudflare account, check its bot settings and AI Crawl Control. Settings such as Bot Fight Mode can challenge automated traffic broadly, and Cloudflare domains created since July 2025 block known AI crawlers by default.
  3. For Akamai, DataDome, HUMAN, Imperva or Kasada, ask the provider or your developer to let verified bots and the AI crawlers you want through, instead of challenging all automated traffic.

Check it yourself

Fetch a product page the way our crawler does and see what comes back:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GlassAisleBot/1.0; +https://glassaisle.com/bot)" https://yourstore.com/products/your-product

A 200 is a pass. A 403, a 429 or a 503 is what agents meet too.

See how your store does

The free audit runs this check and the rest on your store and emails you your score, your grade and your top findings, with the evidence for each.