Every small business with a website is now quietly making a decision it may not know it is making: which AI crawlers get to read its content, and on what terms. A wave of changes from Cloudflare, which sits in front of a large share of the web, has turned that once-invisible question into a live business choice with real consequences for traffic, visibility, and even revenue.
Why AI crawlers suddenly matter to your bottom line
For years, automated crawlers were something only SEO specialists thought about. That changed as AI assistants became a front door to the web. Cloudflare has responded by splitting AI crawlers into three distinct purposes, and the distinction is the key to making a smart call. Search crawlers collect and index content so an assistant can cite it in an answer later. Agent crawlers act in real time on a person’s behalf, such as fetching a page a user just asked about. Training crawlers pull content to train or fine-tune a model, with no direct benefit flowing back to you.
Those three uses have very different value to a small business. A search or agent visit can surface your brand inside ChatGPT, Perplexity, or Google’s AI overviews and send a customer your way. A training scrape mostly gives your words to a model for free.
The September 15 default that could catch you off guard
Here is the change owners need to circle on the calendar. Starting September 15, 2026, Cloudflare’s default settings will block Training and Agent crawlers on any page that displays ads, while leaving Search allowed. Crucially, these new defaults apply to new customers, newly added sites, and all existing free-plan customers, which describes a very large number of small businesses.
The upside is protection you did not have to configure. The risk is the opposite problem reported across the industry: many small businesses on Cloudflare may end up blocking useful AI crawlers without ever deciding to. If your content stops appearing in AI answers this autumn and you never touched a setting, this is the first place to look.
Cloudflare’s bigger bet: getting paid, not just blocking
Blocking is only half the story. Cloudflare is evolving its Pay Per Crawl marketplace into a Pay Per Use model, designed to charge AI companies when your content actually creates value rather than merely when a bot fetches it. Through partnerships with search platforms such as Ceramic.ai and You.com, the goal is for publishers to earn when their material shows up in an AI-generated result. For most small firms this is early and aspirational, but it signals the direction of travel: your content is an asset that can be gated, licensed, or monetised, not just given away.
What small business owners should actually do
The wrong move is a reflexive block-everything switch. Blanket blocking protects content but can quietly erase the AI-driven referrals that are becoming a genuine traffic source. The nuanced approach most experts now recommend is to separate the crawler types and match each to your goals.
Keep search and citation crawlers open if visibility matters to you, which for a growing consultancy or local business it almost always does. Selectively block training-only crawlers, such as GPTBot or Google-Extended, if you would rather not hand your material to model training for nothing. And most important, audit what your current setup is doing today rather than assuming the defaults match your intent. Treat it the same way you would any other operational system, with a clear owner and a periodic review, much as you would when rolling out AI workflow automation across your back office.
Visibility inside AI answers is fast becoming its own marketing channel, sitting alongside the paid and organic reach you already manage through tools like AI-assisted marketing platforms. The businesses that win will not be the ones that panic and slam the door, nor the ones that leave it wide open by accident. They will be the ones that decide, on purpose, exactly which AI crawlers they let in and why.
