---
description: Monitor and control how AI services access your website content.
title: AI Crawl Control
image: https://developers.cloudflare.com/og-docs.png
---

[Skip to content](#main-content)

> Documentation Index  
> Fetch the complete documentation index at: https://developers.cloudflare.com/ai-crawl-control/llms.txt  
> Use this file to discover all available pages before exploring further.

# AI Crawl Control

Last updated Aug 25, 2026|Copy as Markdown|[View as Markdown](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/index.md)|[Agent setup](https://4f1c5cd2.previews.developers.cloudflare.com/agent-setup/)

Available on all plans

Monitor and control how AI services access your website content.

AI companies use web content to train their models and power AI applications. AI Crawl Control (formerly AI Audit) gives you visibility into which AI services are accessing your content, and provides tools to manage access according to your preferences.

With AI Crawl Control, you can:

* **See which AI services access your content** \- Monitor the dashboard to see crawler activity and request patterns
* **Control access with granular policies** \- Set allow or block rules for individual crawlers
* **Monitor robots.txt compliance** \- Track which crawlers follow your directives and create enforcement rules
* **Explore monetization options** \- Set up pay per crawl pricing for content access [(private beta)](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/)
* **Deploy with zero configuration** \- Works automatically on all Cloudflare plans
[Get started ](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/get-started/) 

---

## Features

[Manage AI crawlers](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/)

Control how AI crawlers interact with your domain.

Manage AI crawlers

[Analyze AI traffic](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/)

Gain insight into how AI crawlers are interacting with your pages.

Analyze AI traffic

[Track robots.txt](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/features/track-robots-txt/)

Track the health of `robots.txt` files and identify which crawlers are violating your directives.

Track robots.txt

[Pay Per Crawl](https://4f1c5cd2.previews.developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/)

Allow AI crawlers to access content by paying per crawl.

Pay per crawl

---

## Use cases

### Publishers and content creators

Publishers and content creators can monitor which AI crawlers are accessing their articles and educational content. Set policies to allow beneficial crawlers while blocking others.

### E-commerce and business sites

E-commerce and business sites can identify AI crawler activity on product pages and business information. Control access to sensitive data like pricing and inventory.

### Documentation sites

Documentation sites can track how AI crawlers are accessing their technical documentation. Gain insight into how AI crawlers are engaging with your site.

---

## Related Products

[Bots](https://4f1c5cd2.previews.developers.cloudflare.com/bots/)

Identify and mitigate automated traffic to protect your domain from bad bots.

[Web Application Firewall](https://4f1c5cd2.previews.developers.cloudflare.com/waf/)

Get automatic protection from vulnerabilities and the flexibility to create custom rules.

[Analytics](https://4f1c5cd2.previews.developers.cloudflare.com/analytics/)

View and analyze traffic on your domain.

Was this helpful?

YesNo

## On this page

[![](https://4f1c5cd2.previews.developers.cloudflare.com/_astro/logo.te5VL_aD.svg)Docs](https://4f1c5cd2.previews.developers.cloudflare.com/)

```json
{"@context":"https://schema.org","@type":"WebPage","@id":"https://developers.cloudflare.com/ai-crawl-control/#page","headline":"Overview · Cloudflare AI Crawl Control docs","description":"Monitor and control how AI services access your website content.","url":"https://developers.cloudflare.com/ai-crawl-control/","inLanguage":"en","image":"https://developers.cloudflare.com/og-docs.png","dateModified":"2026-08-25","publisher":{"@type":"Organization","name":"Cloudflare","description":"One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure","url":"https://www.cloudflare.com/","sameAs":["https://github.com/cloudflare","https://www.linkedin.com/company/cloudflare","https://x.com/cloudflare"],"logo":{"@type":"ImageObject","url":"https://developers.cloudflare.com/logo.svg"},"address":{"@type":"PostalAddress","streetAddress":"101 Townsend St","addressLocality":"San Francisco","addressRegion":"CA","postalCode":"94107","addressCountry":"US"},"contactPoint":[{"@type":"ContactPoint","contactType":"Customer Support","url":"https://support.cloudflare.com/","availableLanguage":["English"]},{"@type":"ContactPoint","contactType":"Sales","url":"https://www.cloudflare.com/contact/","availableLanguage":["English"]}]},"isPartOf":{"@type":"WebSite","@id":"https://developers.cloudflare.com/#website","name":"Cloudflare Docs","url":"https://developers.cloudflare.com/"},"keywords":["AI"]}
```