Your firewall is hiding you from ChatGPT

I ran a crawl on a new client’s site. A standard scan, nothing exotic: structure, status codes, titles, canonicals, the minimum to understand what I was dealing with.

The crawler stopped almost immediately. A handful of URLs collected, then nothing but errors.

In a browser, the site worked perfectly.

The culprit was the firewall

I went to see what was running on the WordPress installation and found Wordfence, one of the most widely used security plugins in the world. Nobody had put it there to cause harm. Years ago, someone had installed it and configured it aggressively to stop scrapers and login attempts, and nobody had touched it since.

Wordfence blocks non-human traffic in several ways: limits on requests per minute, behavioral rules (too many pages, too many 404s), blocks on user agent patterns, proper firewall rules. In Extended Protection mode it runs before WordPress loads, so the request is turned away before it even touches the site.

In the list of services Wordfence exempts by default you’ll find Google, Sucuri, Uptime Robot, StatusCake, Seznam. There is no AI provider. Verified Google crawlers get through because the plugin does a reverse DNS check and recognizes them. Everything else is “just another bot”.

My crawler was just another bot. So far, no big deal: I change the user agent, slow down, finish the audit.

The real question: if it blocks me, who else does it block?

The interesting part came afterwards. If that firewall stops an SEO crawler, what happens when an AI assistant tries to open the site?

It’s not a theoretical question. A growing share of people no longer reach a site by starting from a search results page. They ask ChatGPT, Perplexity, Gemini, and the model goes and reads the pages in real time to answer. If the page won’t open, the model doesn’t say “the site is protected”. It says something else, or it talks about someone else.

I ran a test. I asked an agent to browse the site and report back what was on a few specific pages. It couldn’t.

I turned Wordfence off and ran the exact same test again. The agent browsed the site without any trouble.

Not all AI bots are the same

Here we need a distinction that almost nobody makes, and it’s the reason so many sites cut themselves off without meaning to.

The AI bots that visit a site fall into three different categories, with different consequences:

1. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). They collect content to train models. Blocking them is a legitimate choice. A publisher who lives off original content has every reason to do it, and isn’t losing traffic today.

2. Indexing crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot). They build the index that assistants consult when they answer. Blocking them means not showing up in the answers. It’s the equivalent of putting noindex on Google, except nobody sends you a notification.

3. Real-time fetchers (ChatGPT-User, Perplexity-User, Claude-User). They kick in when a person asks something that concerns that site. They’re the most valuable category by far: a live user, with a stated intent, is asking specifically about you. Blocking these is the digital version of slamming the door in the face of someone who just rang the bell.

A hand-configured firewall makes no such distinction. It sees non-human traffic and treats it all the same. It’s the same friction I wrote about with WordPress and agentic tools: the stack was born when visitors were only people with a browser, and every piece assumes that is still the case. The client thinks they’ve blocked scrapers, and in fact they’ve blocked their potential customers too.

The problem is that you can’t see it anywhere

This is the part that convinced me to write about it.

A block like this leaves no trace where you’d go looking. Analytics records nothing, because the request is turned away before any script loads. Search Console tells you nothing, because Googlebot passes through just fine. The site is perfect in a browser, so the client has no reason to suspect anything. The only traces are in Wordfence’s Live Traffic, inside a panel nobody has opened in years.

You can stay invisible to AI agents for months without the slightest signal that it’s happening. And if you work with a vendor who only handles content or only campaigns, nobody will ever look for that signal.

Turning off the firewall is not the solution

Let me be clear: I switched Wordfence off to run a diagnosis, not to leave it off. A WordPress site without protection doesn’t last long, and I’ve already written about what a compromised site looks like and what it costs to get it back on its feet. The right move isn’t to remove the firewall, it’s to teach it to tell the difference.

On a site with this problem, this is the order I follow:

• Live Traffic, filtered on blocks. Wordfence always records the reason. You need to know whether it blocked because of a request limit, a user agent pattern or a firewall rule, because the fix is different in each case.
• Firewall > Blocking. This is where custom patterns live, often added years earlier by someone who wanted to stop one specific scraper and wrote a rule that was too broad.
• Rate Limiting. Many default values are set to unlimited, so if the site is blocking, someone tightened it by hand. Generous limits with a “throttle” action are better than strict limits with a “block” action.
• robots.txt. It’s common to find Disallow rules on AI bots left by an SEO plugin or a previous vendor, maybe from an option switched on once and never reviewed.
• The layer above WordPress. If the site is on managed hosting or behind Cloudflare, the block can happen at the edge, before the request reaches the plugin. In that case there is nothing in Wordfence’s logs and you’re looking in the wrong place.
• Retest with a real agent. Not with a tool that claims to simulate one. Ask an assistant to read you a specific page and see what it says.

And finally the decision on the merits, which isn’t technical: which bots do you want to let in. Blocking training while letting indexing and real-time fetching through is a defensible position for almost everyone. Blocking everything indiscriminately isn’t a position, it’s a configuration nobody has ever reviewed.

The moral

None of these sites is “broken”. They work, they load, they convert. They simply exist for humans with a browser and don’t exist for a growing share of the people looking for information.

These are things you only see if someone looks at the site as a whole, server included, and keeps looking after launch. A configuration made in 2021 for a 2021 problem doesn’t update itself when the way people search changes.

On this site, in the end, I reconfigured the firewall. On others the conclusion was different: if the layer that blocks you exists only because there’s a WordPress to defend underneath, at some point it makes sense to change the engine without changing the bodywork. A static build has no surface to protect, so it has no gatekeeper that gets it wrong when someone knocks.

Want to know if your site is visible to AI assistants?

Tell us about your project and we’ll take a look together: zeromega.it/en/contact.