· web scraping

Web scraping by AI crawlers: your policy is whatever your site enforces

Web scraping used to mean a competitor copying prices. In 2026 it means AI crawlers harvesting your content for training and answers, at a scale most site owners have never looked at. A study of 924 small-business websites (LovedByAI, September to October 2026) counted a median 1,050 AI-crawler requests per site per month, about 35 a day, from 11 different AI crawlers.

AgentBlock is Ingenious AI's one-click answer. It blocks known AI agents with a Cloudflare Worker, a WordPress plugin or a JS snippet, with no user accounts to manage. This check scores the gap between the policy you think you have and the one your site actually enforces.

Honest scope, read this first: this page distinguishes policy from enforcement. A robots.txt line is a request, not a barrier; well-behaved crawlers honor it and scrapers ignore it. AgentBlock classifies known agent user-agents and blocks at the edge. It does not stop a scraper that lies about its identity, and no tool honestly can promise that.

Web scraping at scale: what the 924-site study found

The LovedByAI study read a month of server logs from 924 small-business websites, the kind of sites that assume they are beneath notice. AI crawlers made 40.7% of identified bot page requests, level with all search engines combined at 40.8%. On 79% of the sites, AI crawlers out-requested Googlebot, at a median ratio of 3.1 AI-crawler requests for every one from Google.

The most active single crawler was Meta-ExternalAgent at 21.6% of AI-crawler requests, ahead of ClaudeBot at 18.5% and GPTBot at 12.2%. None of this appears in ordinary analytics, because bots do not run the tracking script. The scraping is visible only in server logs, which is why most owners have a policy about it that their site has never enforced.

1,050median AI-crawler requests per small site per month (LovedByAI, 924 sites)
40.7%of identified bot requests were AI crawlers, level with search engines at 40.8%
21.6%share of the most active AI crawler, Meta-ExternalAgent
11different AI crawlers visiting the median site in one month

Why does robots.txt not stop AI scraping?

Because robots.txt is a convention, not a control. It publishes your wishes in a file honest crawlers read and honor. A training scraper that ignores it loses nothing: the file cannot refuse a request, throttle it or even reliably notice it. Policy lives in robots.txt; enforcement has to live at the edge, in the path of the request itself.

Check your scraping policy: wishes or enforcement?

Answer for your main site as it behaves today. Everything runs in your browser; nothing is sent anywhere.

From wishes to enforcement: the working order

  1. Read the logs first: name your top AI crawlers and their volumes for the last month. Policy written blind is policy written wrong.
  2. Decide in three buckets: training crawlers, AI search crawlers and transactional agents get separate answers. One blanket rule serves none of them well.
  3. Publish the policy: robots.txt carries the public statement, so honest agents can comply without being blocked.
  4. Enforce at the edge: classify known agent user-agents and block the denied ones in the request path. AgentBlock ships this as a Worker, a plugin or a snippet.
  5. Review monthly: new crawlers appear and volumes shift. A quarterly look at the logs keeps the policy a fact.

AgentBlock, by Ingenious AI, gives you one-click blocking of known AI agents through a Cloudflare Worker, WordPress plugin or JS snippet: pick a policy, download the kit, and the edge enforces it, with no accounts to create and nothing to babysit. See how AgentBlock works.

FROM INGENIOUS AI
AgentBlock turns your scraping policy into enforcement in one click, on Cloudflare, WordPress or a plain snippet.
All 21 platforms: DisclosureGuard  ·  ModelMinder  ·  TermsWatch  ·  Vibe-Code Audit  ·  Rogue-Agent Defense  ·  BreachHunter  ·  ChatBreaker  ·  Agent Flight Recorder  ·  Agent Auth Gateway  ·  DomainGuard  ·  ContentCraft AI  ·  EvalScope  ·  BoundaryVault  ·  Agentic Receipts  ·  Agent-Ready Store Audit  ·  AgentBlock  ·  HomeShield MCP  ·  AgentGate  ·  ADMT  ·  DueLadder  ·  WireWatch
One suite, one deal: the Ingenious AI Security Suite. All 21 platforms are live now:
Chat with us on WhatsApp →

Quiz: how well do you know the AI scraping wave?

1. How many AI-crawler requests hit the median small site per month?

2. Which AI crawler was most active in the 924-site study?

3. Why does AI scraping not appear in your analytics?

4. What share of identified bot requests were AI crawlers?

5. Why does robots.txt fail against a determined scraper?

Frequently asked questions

What is AI web scraping?

AI web scraping is the automated collection of site content by AI companies' crawlers, mainly to train models and to answer questions in AI search. It runs at industrial scale: a study of 924 small-business sites counted a median 1,050 AI-crawler requests per site per month in autumn 2026. AgentBlock exists to let site owners decide which of those agents get served.

What is AgentBlock?

AgentBlock is Ingenious AI's one-click blocking kit for known AI agents. You pick a policy and deploy it as a Cloudflare Worker, a WordPress plugin or a JS snippet. It classifies known agent user-agents and blocks the ones your policy denies. By design it has no user accounts: the policy lives in your own edge, not in our dashboard.

Does robots.txt block AI crawlers?

robots.txt declares your preferences; it cannot enforce them. Honest crawlers from major AI companies generally honor it, and it is worth publishing as your public policy. Scrapers that ignore it lose nothing, which is why enforcement at the edge, classifying user-agents in the request path, is the layer that makes the policy real.

Which AI crawlers visit a typical site?

In the LovedByAI study of 924 small-business sites (September-October 2026), the most active was Meta-ExternalAgent at 21.6% of AI-crawler requests, then ClaudeBot at 18.5% and GPTBot at 12.2%. OpenAI's crawlers together took the largest company share at 27.2%. The median site saw 11 different AI crawlers in a month.

Should I block all AI crawlers?

Not necessarily, and AgentBlock does not force one answer. Training crawlers take content and return nothing; AI search crawlers can send visitors; transactional agents may become customers. Most owners land on a split policy: block training, allow search, decide agents case by case. What matters is that the decision is yours and enforced.

Sources: LovedByAI, AI Crawler Statistics 2026 (924 small-business websites, September 6 - October 5, 2026) · Cloudflare Radar AI-crawler measurements (2026) · Semrush keyword data (web scraping, 1,830,000/mo global).

Related checks