GlassAisleBot

How it identifies itself

Every request carries this user agent and a From header with our contact address.

User-Agent: Mozilla/5.0 (compatible; GlassAisleBot/1.0; +https://glassaisle.com/bot)
From: bot@glassaisle.com

Pages are also loaded in a headless browser on Cloudflare Browser Run, which sends the same user agent and From header.

What it fetches

• robots.txt, which it reads first and always follows.
• The sitemap, the homepage, a few collection pages and up to 5 product pages.
• Policy, FAQ, contact and about pages linked from the homepage.
• llms.txt and the agent profile at /.well-known/ucp, if the store has them.

It never submits a form, signs in, accepts a cookie banner or solves a challenge. In a full audit the store owner has authorized, it also opens one cart link to check that checkout loads. It never places an order.

How fast it goes

• At most 2 requests at a time, at least 1 second apart.
• At most 40 fetches and 10 page loads in one audit, and it stops after 6 minutes.
• At most 2 audits and 120 requests per store per day.
• It honors Crawl-delay in robots.txt.

When a site turns it away

If your bot protection blocks it 2 times, or answers with 429, it stops asking your site for the rest of that audit and honors Retry-After. It never rotates IP addresses, pretends to be another crawler or a browser, or works around a block.

How to opt out

Add this to your robots.txt and GlassAisleBot will not fetch any page on your site:

User-agent: GlassAisleBot
Disallow: /

To keep it off only some paths, list them with Disallow lines under the same User-agent line. It follows the group for GlassAisleBot if your robots.txt has one, and the group for all crawlers otherwise.

Contact

Questions or problems with the crawler go to bot@glassaisle.com.