ClaudeBot is Anthropic’s web crawler; if you saw it in your logs, verify the request, decide your crawler policy, and use robots.txt or access controls as needed, with broader context in our Claude AI guide.
c-ai.chat is an independent guide to Claude AI. We are not Anthropic, and we do not operate claude.ai.
- The short answer
- Why ClaudeBot appears in logs
- What to do next
- FAQ: other questions readers ask
- The practical verdict
- Sources
The short answer
ClaudeBot is the crawler associated with Anthropic, the company that makes Claude. It is not the Claude chat product. If ClaudeBot appears in your server logs, an automated crawler likely requested a page on your site.
- What it is: Anthropic’s web crawler.
- What it is not: The Claude chatbot at claude.ai.
- Main control: robots.txt for cooperative crawler access.
- Security rule: Use authentication for private content.
- Official references: Anthropic support, docs, trust, and status domains.
The distinction matters. Claude chat runs at claude.ai. ClaudeBot is a crawler name that website owners may see in logs, CDN dashboards, firewall reports, or analytics tools.
For official guidance, check Anthropic’s own sites: support.anthropic.com, anthropic.com, docs.claude.com, and platform.claude.com.
Why ClaudeBot appears in logs
Most people search for ClaudeBot after seeing it in access logs. The practical question is usually simple: is it official, should I allow it, and how do I control it?
Do not confuse ClaudeBot with Claude, third-party bots named “Claude,” or API traffic from your own application. If you are building with Anthropic’s API, use our Claude API guide. If you want to understand the user-facing assistant, see our Claude features overview and Claude models guide.
For site owners, the main public control is robots.txt. It tells cooperative crawlers which paths they should not access. It is not a security system. Anything private should sit behind authentication, not behind a robots.txt rule.
| What you saw | What it likely means | What to check |
|---|---|---|
| ClaudeBot in server logs | Anthropic’s crawler requested a URL | User agent, request path, response code, volume, robots.txt |
| claude.ai in a browser | A person is using the official Claude product | Account settings, plan limits, product documentation |
| Claude API traffic | An app is calling Anthropic’s API | API keys, usage dashboard, billing settings |
| A “Claude bot” in another app | A third-party or unofficial integration may be involved | App permissions, developer identity, connected accounts |
ClaudeBot crawling your public site is also different from billable Claude API usage. API costs apply when your account sends requests to Anthropic’s API. For subscription and API cost context, see our Claude pricing guide.
What to do next
If you found ClaudeBot in your logs, verify the traffic first. Then decide whether to allow, limit, or block it.
Check the request details
Review the user agent, request path, timestamp, response code, IP pattern, and request volume. A single request to a public page is different from repeated requests to sensitive paths.
Review Anthropic’s guidance
Use Anthropic’s own support and documentation domains for crawler policy details. Start with support.anthropic.com and docs.claude.com.
Set your access rule
If you want to block ClaudeBot, add a robots.txt rule for
ClaudeBot. If you only want to block selected areas, disallow those paths instead of the whole site.Test the deployed file
Confirm that
/robots.txtis reachable, returns a normal success response, and contains the exact rule you intended.Protect private content properly
Use login, permissions, firewall rules, or other access controls for non-public material. Do not rely on robots.txt for privacy.
Example: A site-wide block tells a cooperative crawler not to request any path on your site.
User-agent: ClaudeBot
Disallow: /Example: A narrower rule can allow public pages while excluding account, checkout, or internal sections.
User-agent: ClaudeBot
Disallow: /account/
Disallow: /checkout/
Disallow: /internal/If you use a CDN, web application firewall, or managed host, check whether it caches robots.txt. A stale cached file can make a correct update look broken. If traffic volume looks unusual, check status.claude.com and your own infrastructure logs before deciding what changed.
FAQ: other questions readers ask
Is ClaudeBot safe?
ClaudeBot is associated with Anthropic, but you should still verify traffic instead of trusting a user-agent string alone. User agents can be spoofed. Check the request pattern, volume, paths, and Anthropic’s official guidance.
Can I block ClaudeBot?
Yes. Site owners can use robots.txt to tell cooperative crawlers not to access selected paths or the whole site. For a full block, use User-agent: ClaudeBot with Disallow: /.
Is ClaudeBot the same as Claude?
No. Claude is the AI assistant made by Anthropic and available through claude.ai, apps, and API access. ClaudeBot is a crawler name seen by website operators.
Does blocking ClaudeBot remove my content from Claude?
Blocking future crawler access is not the same as deleting data from every system. For formal privacy, trust, or data-handling questions, use Anthropic’s official channels, including trust.anthropic.com and support.anthropic.com.
Will ClaudeBot cost me money?
ClaudeBot requests can use server bandwidth, like other crawler traffic. They are not the same as Claude API calls billed to your Anthropic account. If you pay for Claude through the API, review usage in the Anthropic platform and see our Claude FAQ for common account questions.
Should publishers allow ClaudeBot?
That depends on your content policy. Some publishers allow AI crawlers to access public pages. Others block them because of licensing, attribution, or commercial concerns. Decide at the site-policy level, then implement the rule clearly.
Does robots.txt protect private pages?
No. robots.txt is a crawler instruction. It does not stop a browser, logged-in user, misconfigured app, or non-cooperative bot from reaching a URL. Use authentication and permissions for private pages.
The practical verdict
ClaudeBot is a normal item to investigate if you run a website. Seeing it in logs does not automatically mean something is wrong. It means you should apply the same checks you would use for any major crawler: verify the traffic, set a clear policy, publish accurate robots.txt rules, and secure private content properly.
Reasons to allow it
- Your content is public and intended for broad discovery.
- You want AI systems to understand public information about your site.
- You already allow comparable crawlers under your content policy.
Reasons to block or limit it
- Your content is licensed, gated, or commercially sensitive.
- Your publisher policy restricts AI crawler access.
- You want tighter control over automated access to specific sections.
The right answer depends on your site, your licensing model, and your tolerance for automated access. A narrow robots.txt rule is often better than a broad block if only some paths are sensitive.
Independent guide. Not affiliated with Anthropic. For the official Claude product, visit claude.ai.
Last updated: 2026-05-12
