AI Crawler Access: Check robots.txt, Firewalls and Search Visibility Separately

A public website needs to be accessible for a search service to retrieve its pages. That is one part of visibility, not a promise that the service will recommend your business. Check your crawler policy, the site's actual responses and observed search results as separate questions.

Identify the crawler and its purpose

OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for content that may be used in model training. Their controls are independent. ChatGPT-User handles certain user-requested visits and has different behavior. Blocking a search crawler is not the same action as opting out of training.

Anthropic distinguishes ClaudeBot, Claude-SearchBot and Claude-User. Perplexity likewise distinguishes PerplexityBot from its user-triggered fetcher. Check each provider's current documentation instead of applying one rule to every product.

Google's Google-Extended control covers specified Gemini training and grounding uses. It does not control inclusion or ranking in Google Search. Do not describe every AI-related control as training-only, or assume a new opt-out erases previously collected data.

Review rules without discarding existing restrictions

Read the robots.txt file on the actual production hostname. Identify the group applicable to each crawler and the paths it allows or disallows. If adding a crawler-specific group, check that you retain any intended path restrictions rather than copying an unrestricted example over them.

Keep administrative and private content protected by appropriate access controls. robots.txt communicates crawling preferences; it is not authentication. Make search-access changes deliberately and preserve the site's existing security requirements.

Check the firewall independently

An Allow rule does not override a CDN or web application firewall. A request can be denied or challenged before it reaches your website. Inspect the response and relevant security events when investigating access.

Use provider documentation to identify legitimate crawler traffic. Where a provider publishes IP ranges, check those alongside the request details. A user-agent name alone can be imitated. Avoid opening access to every request that merely claims to be a search bot.

Keep evidence at the right level

• Rule review: Shows what the published crawling instructions say for the relevant path.
• HTTP response check: Shows what happened to that particular request. A test using a crawler name does not reproduce the provider's actual network origin.
• Verified server or CDN logs: Can show whether an identified crawler requested a page and what response it received.
• Search observation: Records whether a particular answer included your page. It does not establish when or how the provider obtained the content.

Save the date, URL, request details and result of each check. If your site does not appear in an answer, investigate access alongside relevance and indexing rather than declaring a firewall failure. If it does appear, record the citation without treating it as a guarantee of future placement.

Resolve demonstrated access problems first. Additional files or markup should have a documented purpose; they cannot substitute for accurate public content and a working website.