How to Verify Googlebot and Catch Fake AI Crawlers

A user agent is a claim, not proof. Anyone can send Googlebot/2.1 or GPTBot/1.4 with curl -A. Scrapers do this because many sites whitelist Googlebot, and because a blocklist keyed on a user agent stops nobody who doesn’t use that user agent. Common Crawl says it plainly on its CCBot page: “we are aware of crawlers falsely identifying themselves as CCBot.”

The numbers are not small:

Centinel Analytica’s “Fake Googlebot and AI crawlers: 2026 data” (22 September 2026) covers news and media sites it protects, from late June to 21 September 2026. On the median site, 1 in 7 requests claiming to be Googlebot failed verification, as did 84% of GPTBot claims and 99% of Perplexity-User claims. That’s one vendor and one industry, so treat it as a warning sign, not a universal rate.
A single-site log study by ROI & Shine (May 2026) found 18% of “Googlebot” requests were spoofed.
Centinel also found 94% of fake Googlebot requests copied the real, current version string. Checking the version won’t catch them. Only checking the network will.

This guide shows how to find out which “bots” in your nginx logs are real. It uses the two methods the operators themselves document: forward-confirmed reverse DNS and published IP ranges.

The two verification methods

Method	How it works	Who supports it
Forward-confirmed reverse DNS (FCrDNS)	PTR lookup on the IP, check the hostname’s domain, then resolve that hostname and confirm it returns the same IP	Google, Bing, Apple, Common Crawl (IPv4)
Published IP ranges	Match the IP against the operator’s JSON list of CIDR ranges	Google, Bing, OpenAI, Anthropic, Perplexity, Apple, Common Crawl

Most AI crawlers do not set useful reverse DNS, so for them the IP list is the only method. A PTR record on its own proves nothing. Whoever controls an IP block can set its PTR to crawl-1-2-3-4.googlebot.com. That is why the forward lookup is required.

flowchart TD
    A[Log line: UA claims a known bot] --> B{Operator publishes IP ranges?}
    B -- yes --> C{IP inside official CIDR list?}
    C -- yes --> V[Verified]
    C -- no --> D{Operator supports rDNS?}
    B -- no --> D
    D -- no --> F[Unverified / likely fake]
    D -- yes --> E[PTR lookup on IP]
    E --> G{Hostname ends in operator domain?}
    G -- no --> F
    G -- yes --> H[Forward lookup of hostname]
    H --> I{Returns the original IP?}
    I -- yes --> V
    I -- no --> F

Step 1: Find who claims to be Googlebot

This assumes nginx’s default combined log format. Splitting each line on " makes field 6 the user agent:

bash

LOG=/var/log/nginx/access.log
awk -F'"' '$6 ~ /Googlebot/ {split($1,a," "); print a[1]}' "$LOG" \
  | sort | uniq -c | sort -rn > /tmp/googlebot-ips.txt
wc -l /tmp/googlebot-ips.txt
head /tmp/googlebot-ips.txt

To include rotated logs, use zcat -f /var/log/nginx/access.log* in place of "$LOG".

Step 2: Verify with reverse and forward DNS

Google’s documented procedure is to run host on the IP, check that the domain is googlebot.com, google.com or googleusercontent.com, then run host on the hostname and confirm the IP matches. Google’s own example:

text

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Google’s own docs (last updated 20 March 2026) are stricter than that summary, because Google splits its traffic into groups, each with its own reverse DNS pattern and IP file:

Group	Examples	Reverse DNS	IP range file	robots.txt
Common crawlers	Googlebot	`crawl-*.googlebot.com` or `geo-crawl-*.geo.googlebot.com`	`common-crawlers.json`	Respected for automatic crawls
Special-case crawlers	AdsBot	`rate-limited-proxy-***.google.com`	`special-crawlers.json`	May not be respected
User-triggered fetchers	Fetches a person asked for	`*.gae.googleusercontent.com` or `google-proxy-*.google.com`	`user-triggered-fetchers.json`, `user-triggered-fetchers-google.json`	Generally ignored
User-triggered agents	Google-Agent	as above	`user-triggered-agents.json`	Generally ignored

All files live under https://developers.google.com/static/crawling/ipranges/.

A trap that catches people: Google-Extended is not a crawler. Google documents it as a robots.txt token only, with no user agent of its own. Any request whose user agent says Google-Extended is fake by definition. In one published eight-day log sample (aulvem.com, August 2026), all 123 requests claiming to be Google-Extended were unverified, and 70 of them went for paths like .env and wp-login.

The loop below accepts only googlebot.com for a Googlebot user agent:

bash

while read -r count ip; do
  ptr=$(host "$ip" 2>/dev/null | awk '/pointer/ {print $NF}' | sed 's/\.$//' | head -1)
  case "$ptr" in
    *.googlebot.com)
      if host "$ptr" | awk '/has (IPv6 )?address/ {print $NF}' | grep -qx "$ip"; then
        echo "OK    $count $ip $ptr"
      else
        echo "FAKE  $count $ip forward-mismatch $ptr"
      fi ;;
    *) echo "FAKE  $count $ip ptr=${ptr:-none}" ;;
  esac
done < /tmp/googlebot-ips.txt | sort -k2 -rn

For Bing, use the same loop with *.search.msn.com. For Applebot use *.applebot.apple.com, and for CCBot over IPv4 use *.crawl.commoncrawl.org. A caveat on IPv6: host prints addresses in compressed form, and that can differ from how nginx logged them. Use the Java checker below if you see IPv6 false positives.

Step 3: Verify AI crawlers against published IP ranges

This is the official list of IP range files as of October 2026:

User agent token	IP range JSON
Googlebot (common crawlers)	https://developers.google.com/static/crawling/ipranges/common-crawlers.json
bingbot	https://www.bing.com/toolbox/bingbot.json
GPTBot	https://openai.com/gptbot.json
OAI-SearchBot	https://openai.com/searchbot.json
ChatGPT-User	https://openai.com/chatgpt-user.json
ClaudeBot, Claude-User, Claude-SearchBot	https://claude.com/crawling/bots.json (one shared list)
PerplexityBot	https://www.perplexity.com/perplexitybot.json
Perplexity-User	https://www.perplexity.com/perplexity-user.json
CCBot	https://index.commoncrawl.org/ccbot.json

Older guides link to googlebot.json. Google’s current verification page lists common-crawlers.json. If you hardcoded the old URL, check it still resolves.

For a quick one-off check you only need curl, jq and Python’s standard library:

bash

IP=20.171.206.10   # replace with the IP from your log
curl -s https://openai.com/gptbot.json \
 | jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix)' \
 | python3 -c 'import sys,ipaddress as i; a=i.ip_address(sys.argv[1]); print(any(a in i.ip_network(l.strip()) for l in sys.stdin if l.strip()))' "$IP"

True means the IP is in OpenAI’s GPTBot list. False on a request whose user agent says GPTBot means it is spoofed.

Prefer not to run anything yet? Paste a sample of your log into the free nginx log checker. It runs in your browser and checks crawler IPs against these lists.

Step 4: Check every bot in the log at once (Java)

Checking IPs one at a time doesn’t scale past a handful. The single-file program below reads a combined-format log on stdin and loads each operator’s official ranges. It verifies every claimed bot (IP list first, FCrDNS as fallback) and prints verified and fake counts plus the noisiest fake sources. It needs Java 17+ and no dependencies.

java

// BotVerify.java - run: java BotVerify.java < /var/log/nginx/access.log
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.math.BigInteger;
import java.net.InetAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class BotVerify {

    record Bot(String token, List<String> jsonUrls, List<String> rdnsSuffixes) {}

    record Cidr(BigInteger network, BigInteger mask, int byteLen) {
        static Cidr parse(String cidr) throws Exception {
            String[] p = cidr.trim().split("/");
            byte[] addr = InetAddress.getByName(p[0]).getAddress();
            int bits = addr.length * 8;
            int prefix = Integer.parseInt(p[1]);
            BigInteger all = BigInteger.ONE.shiftLeft(bits).subtract(BigInteger.ONE);
            BigInteger hostBits = BigInteger.ONE.shiftLeft(bits - prefix).subtract(BigInteger.ONE);
            BigInteger mask = all.xor(hostBits);
            return new Cidr(new BigInteger(1, addr).and(mask), mask, addr.length);
        }
        boolean contains(byte[] ip) {
            return ip.length == byteLen && new BigInteger(1, ip).and(mask).equals(network);
        }
    }

    // Order matters: more specific tokens before shorter ones.
    static final List<Bot> BOTS = List.of(
        new Bot("Googlebot", List.of("https://developers.google.com/static/crawling/ipranges/common-crawlers.json"), List.of(".googlebot.com")),
        new Bot("bingbot", List.of("https://www.bing.com/toolbox/bingbot.json"), List.of(".search.msn.com")),
        new Bot("OAI-SearchBot", List.of("https://openai.com/searchbot.json"), List.of()),
        new Bot("ChatGPT-User", List.of("https://openai.com/chatgpt-user.json"), List.of()),
        new Bot("GPTBot", List.of("https://openai.com/gptbot.json"), List.of()),
        new Bot("Claude-SearchBot", List.of("https://claude.com/crawling/bots.json"), List.of()),
        new Bot("Claude-User", List.of("https://claude.com/crawling/bots.json"), List.of()),
        new Bot("ClaudeBot", List.of("https://claude.com/crawling/bots.json"), List.of()),
        new Bot("Perplexity-User", List.of("https://www.perplexity.com/perplexity-user.json"), List.of()),
        new Bot("PerplexityBot", List.of("https://www.perplexity.com/perplexitybot.json"), List.of()),
        new Bot("Applebot", List.of(), List.of(".applebot.apple.com")),
        new Bot("CCBot", List.of("https://index.commoncrawl.org/ccbot.json"), List.of(".crawl.commoncrawl.org"))
    );

    static final Pattern LOG = Pattern.compile(
        "^(\\S+) \\S+ \\S+ \\[[^\\]]+\\] \"[^\"]*\" \\d{3} \\S+ \"[^\"]*\" \"([^\"]*)\"");
    static final Pattern PREFIX = Pattern.compile("\"ipv[46]Prefix\"\\s*:\\s*\"([^\"]+)\"");
    static final HttpClient HTTP = HttpClient.newBuilder()
        .connectTimeout(Duration.ofSeconds(10))
        .followRedirects(HttpClient.Redirect.NORMAL).build();
    static final Map<String, List<Cidr>> RANGES = new HashMap<>();
    static final Map<String, Boolean> CACHE = new HashMap<>();

    public static void main(String[] args) throws Exception {
        for (Bot bot : BOTS) {
            for (String url : bot.jsonUrls()) {
                if (RANGES.containsKey(url)) continue;
                List<Cidr> list = new ArrayList<>();
                try {
                    HttpRequest req = HttpRequest.newBuilder(URI.create(url))
                        .timeout(Duration.ofSeconds(20)).build();
                    String body = HTTP.send(req, HttpResponse.BodyHandlers.ofString()).body();
                    Matcher m = PREFIX.matcher(body);
                    while (m.find()) list.add(Cidr.parse(m.group(1)));
                } catch (Exception e) {
                    System.err.println("WARN could not load " + url + ": " + e.getMessage());
                }
                System.err.printf("loaded %d prefixes from %s%n", list.size(), url);
                RANGES.put(url, list);
            }
        }

        Map<String, int[]> stats = new TreeMap<>();      // token -> [verified, fake]
        Map<String, Integer> fakeSources = new HashMap<>();
        BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
        String line;
        while ((line = in.readLine()) != null) {
            Matcher m = LOG.matcher(line);
            if (!m.find()) continue;
            String ip = m.group(1);
            String ua = m.group(2).toLowerCase(Locale.ROOT);
            for (Bot bot : BOTS) {
                if (!ua.contains(bot.token().toLowerCase(Locale.ROOT))) continue;
                boolean ok = CACHE.computeIfAbsent(bot.token() + "|" + ip, k -> verify(bot, ip));
                stats.computeIfAbsent(bot.token(), k -> new int[2])[ok ? 0 : 1]++;
                if (!ok) fakeSources.merge(bot.token() + " " + ip, 1, Integer::sum);
                break;
            }
        }

        System.out.printf("%-18s %10s %10s%n", "claimed bot", "verified", "FAKE");
        stats.forEach((k, v) -> System.out.printf("%-18s %10d %10d%n", k, v[0], v[1]));
        System.out.println("\nTop fake sources (claimed bot, ip, requests):");
        fakeSources.entrySet().stream()
            .sorted((a, b) -> b.getValue() - a.getValue()).limit(25)
            .forEach(e -> System.out.println("  " + e.getKey() + " " + e.getValue()));
    }

    static boolean verify(Bot bot, String ip) {
        try {
            byte[] addr = InetAddress.getByName(ip).getAddress();
            for (String url : bot.jsonUrls())
                for (Cidr c : RANGES.getOrDefault(url, List.of()))
                    if (c.contains(addr)) return true;
            if (bot.rdnsSuffixes().isEmpty()) return false;
            String host = InetAddress.getByAddress(addr).getCanonicalHostName();
            if (bot.rdnsSuffixes().stream().noneMatch(host::endsWith)) return false;
            for (InetAddress fwd : InetAddress.getAllByName(host))
                if (Arrays.equals(fwd.getAddress(), addr)) return true;
            return false;
        } catch (Exception e) {
            return false;
        }
    }
}

Run it:

bash

zcat -f /var/log/nginx/access.log* | java BotVerify.java

Look at the loaded N prefixes lines on stderr. If a source shows 0, the URL or JSON schema has changed. Every bot that relies on that list will then show as fake, so fix the list before you trust the results.

Two things to know before you trust the output:

Behind Cloudflare or a load balancer, $remote_addr is the proxy, so every check fails. Set set_real_ip_from and real_ip_header in nginx first.
Fakes share real names. In the aulvem sample, the same ChatGPT-User string produced 211 verified and 336 unverified requests. Count verified hits only when you report AI crawler traffic, or the fakes will inflate it.

Step 5: Act on the result

Blocking fakes is safe because no real crawler sends them. The nginx geo module matches CIDR lists natively. Generate an include file from the official JSON:

bash

curl -s https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json \
 | jq -r '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) + " 1;"' > /etc/nginx/bots/openai-ips.conf.new \
 && [ -s /etc/nginx/bots/openai-ips.conf.new ] \
 && mv /etc/nginx/bots/openai-ips.conf.new /etc/nginx/bots/openai-ips.conf

Then, in the http context:

nginx

geo $openai_ip {
    default 0;
    include /etc/nginx/bots/openai-ips.conf;
}
map $http_user_agent $claims_openai {
    default 0;
    "~*(GPTBot|OAI-SearchBot|ChatGPT-User)" 1;
}
map "$claims_openai$openai_ip" $fake_openai {
    default 0;
    "10"    1;   # claims OpenAI, IP not in OpenAI ranges
}

And in each server block:

nginx

if ($fake_openai) { return 403; }

Apply it with sudo nginx -t && sudo systemctl reload nginx. Refresh the IP files from cron and keep the [ -s ... ] guard. An empty include would make every real OpenAI request look fake.

Don’t block the real crawlers by IP if your goal is to opt out. Anthropic warns that blocking its IPs “may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.” Use the IP lists to catch impostors. Use robots.txt or user-agent rules to manage the real bots, as shown in Block AI Training Crawlers, Allow AI Search in nginx.

Doing this across many sites

This works well for one server and one afternoon. It gets harder when you have twelve sites, rotating logs, and IP lists that change without notice. Anthropic’s list is one example: a GitHub mirror that tracks it notes “Anthropic’s feed alone moves regularly”.

I’m building Trafficsort to do this continuously. An open-source agent reads your nginx logs, checks every bot claim against the operators’ published ranges and reverse DNS, and emails you when fake-bot traffic surges. Visitor IPs are hashed on your server before anything is sent. If that sounds useful, join the waitlist.

How to Verify Googlebot and Catch Fake AI Crawlers in Your Logs

The two verification methods

Step 1: Find who claims to be Googlebot

Step 2: Verify with reverse and forward DNS

Step 3: Verify AI crawlers against published IP ranges

Step 4: Check every bot in the log at once (Java)

Step 5: Act on the result

Doing this across many sites