Skip to content
pugdocs

AI crawlers

See which AI products, search engines and SEO tools read your site, and why, with @pug-sh/crawlers in your own request handling and a private key.

See which AI products, search engines and SEO tools read your site, and why: to answer someone’s question, to build a search index, or to train a model.

Crawlers fetch HTML and don’t run JavaScript, so a browser analytics script never sees them. This package runs in your site’s own request handling instead. When a request comes from a known crawler, it sends one small report to Pug after the response. Pug checks the request’s IP against the ranges the crawler’s operator publishes, so a scraper borrowing GPTBot’s name reads as unverified.

It has no dependencies and runs on Cloudflare Workers, Vercel, Next.js, Hono, Bun, Deno, Express and plain Node.

Install

npm install @pug-sh/crawlers

You need a private key (prv_...) from your project’s settings in Pug. Keep it on the server, in an environment variable: it can also read your analytics.

Cloudflare Workers and Pages Functions

import { crawlers } from '@pug-sh/crawlers'

export default {
  async fetch(request, env, ctx) {
    const response = await env.ASSETS.fetch(request)
    crawlers({ apiKey: env.PUG_PRIVATE_KEY }).track(request, response, ctx)
    return response
  },
}

Passing ctx lets the Worker finish the report after the response is sent.

Hono, Bun and Deno

import { crawlers } from '@pug-sh/crawlers'

const pug = crawlers({ apiKey: process.env.PUG_PRIVATE_KEY })

app.use(async (c, next) => {
  await next()
  pug.track(c.req.raw, c.res)
})

On Cloudflare, pass c.executionCtx as the third argument. Elsewhere, see the client’s address.

Next.js and Vercel

// proxy.ts
import { crawlers } from '@pug-sh/crawlers'
import { type NextFetchEvent, type NextRequest, NextResponse } from 'next/server'

const pug = crawlers({ apiKey: process.env.PUG_PRIVATE_KEY! })

export function proxy(request: NextRequest, event: NextFetchEvent) {
  pug.track(request, undefined, event)
  return NextResponse.next()
}

The proxy runs before the page, so these reports carry no status code.

Express and node:http

import { crawlers } from '@pug-sh/crawlers'

app.use(crawlers({ apiKey: process.env.PUG_PRIVATE_KEY }).express())

The middleware reports after the response finishes. With plain node:http, call it at the start of your handler: middleware(req, res).

Any other server

Report a request yourself with one POST, from any language:

curl https://api.pugs.dev/sdk.events.v1.EventsService/BatchCreate \
  -H 'x-api-key: prv_...' \
  -H 'content-type: application/json' \
  -d '{"events": [{"eventId": "0193b2c4-8f1e-7a2b-9c3d-4e5f6a7b8c9d", "occurTime": "2026-11-14T09:30:00Z",
       "kind": "crawler_request",
       "client": {"userAgent": "... GPTBot/1.2 ...", "ip": "20.171.207.5"},
       "autoProperties": {"$url": {"stringValue": "https://example.com/pricing"},
                          "$status_code": {"intValue": 200}}}]}'

One call takes up to 1,000 requests. Make eventId (a UUID) once and resend it unchanged on a retry, so Pug stores the request once.

Options

Option Default What it does
apiKey required A private key, prv_....
endpoint https://api.pugs.dev Where Pug runs. A self-hosted install sets its own.
categories all five Which kinds of crawler to report: answers, search, training, seo, other.
skip none Agents to leave out, by name, such as ['Googlebot'].
includeAssets false Also report requests for scripts, styles, images and fonts.
getIP the platform’s Returns the crawler’s address for a request.

The categories

Category Meaning Examples
answers Fetched live because a person asked an AI product ChatGPT-User, Claude-User, Perplexity-User
search Builds a search index that results and AI answers draw on OAI-SearchBot, Googlebot, bingbot
training Collects pages to train models GPTBot, ClaudeBot, CCBot
seo Crawls for an SEO tool AhrefsBot, SemrushBot, MJ12bot
other AI-related, but none of the above OAI-AdsBot, GoogleOther

The AI agents come from Pug’s own table of 35, each with its company and, where the operator publishes them, its IP ranges. Search engines and SEO tools come from the crawler-user-agents list. Both ship inside the package, so a new crawler reaches your site with the package’s next release.

What it sends

Only requests whose user agent names a known crawler, so your visitors’ requests never leave your server. For each one it sends:

Field Value
$url The full URL, without its query string or fragment
client.userAgent The User-Agent header
client.ip The client’s address, for the IP check. Pug never stores it.
$status_code The response’s status, when the adapter sees it

It never sends query strings, cookies, other headers, bodies, requests other than GET and HEAD, or (unless includeAssets is on) static files.

Delivery

One POST per crawler request, after the response, with a 2-second timeout and one retry on a network error or a server error. It never throws, so it can’t break your site, and it logs a failure once per process.

The client’s address

The package reads CF-Connecting-IP on Cloudflare, x-real-ip on Vercel, req.ip on Express (which follows its trust proxy setting) and the socket on plain Node. Pass getIP when that’s wrong:

  • Behind a CDN, your server sees the CDN’s address, so every request fails the IP check. Read the CDN’s client-IP header in getIP.
  • A fetch-style server that Cloudflare or Vercel doesn’t front can’t trust those two headers, since anyone can send them. Return the socket’s address, such as Bun’s server.requestIP(req)?.address.

Running at the edge (Workers, Vercel) sees every request, including those a CDN would answer from its cache.

See them in Pug

Each request is a crawler_request event, tagged as a bot, so it stays out of your other numbers. Insights and the Events list include bots on their own when you ask for crawler_request or one of its properties:

Property Value
$bot_reason The agent’s name, such as GPTBot
$crawler_company Such as OpenAI; none for a search engine or SEO tool from the list
$crawler_category answers, search, training, seo or other
$crawler_verified verified (inside the operator’s ranges), failed (outside them, likely an impostor) or unchecked
$status_code The response’s status, when sent
$url, $pathname, $hostname The page requested

Break crawler_request down by $crawler_category for requests over time, rank $crawler_company or $bot_reason with Top K, or filter $status_code to 404 for the pages crawlers can’t find. A crawler is never listed as a person.

Volume

Crawler requests count toward your Pug usage like any event, and a busy site can get more of them than page views. SEO tools are among the busiest crawlers, so seo is the first category to leave out.

Self-hosted Pug

Point endpoint at your Pug. Its server fetches each operator’s IP ranges when it starts, then every 6 hours. Without internet access, set PUG_CRAWLER_RANGES_ENABLED=false, and every request reads unchecked.

Navigation

Type to search...

↑↓ navigate↵ selectEsc close