See which AI products, search engines and SEO tools read your site, and why: to answer someone’s question, to build a search index, or to train a model.
Crawlers fetch HTML and don’t run JavaScript, so a browser analytics script never sees them. This package runs in your site’s own request handling instead. When a request comes from a known crawler, it sends one small report to Pug after the response. Pug checks the request’s IP against the ranges the crawler’s operator publishes, so a scraper borrowing GPTBot’s name reads as unverified.
It has no dependencies and runs on Cloudflare Workers, Vercel, Next.js, Hono, Bun, Deno, Express and plain Node.
Install
npm install @pug-sh/crawlersYou need a private key (prv_...) from your project’s settings in Pug. Keep it
on the server, in an environment variable: it can also read your analytics.
Cloudflare Workers and Pages Functions
import { crawlers } from '@pug-sh/crawlers'
export default {
async fetch(request, env, ctx) {
const response = await env.ASSETS.fetch(request)
crawlers({ apiKey: env.PUG_PRIVATE_KEY }).track(request, response, ctx)
return response
},
}Passing ctx lets the Worker finish the report after the response is sent.
Hono, Bun and Deno
import { crawlers } from '@pug-sh/crawlers'
const pug = crawlers({ apiKey: process.env.PUG_PRIVATE_KEY })
app.use(async (c, next) => {
await next()
pug.track(c.req.raw, c.res)
})On Cloudflare, pass c.executionCtx as the third argument. Elsewhere, see
the client’s address.
Next.js and Vercel
// proxy.ts
import { crawlers } from '@pug-sh/crawlers'
import { type NextFetchEvent, type NextRequest, NextResponse } from 'next/server'
const pug = crawlers({ apiKey: process.env.PUG_PRIVATE_KEY! })
export function proxy(request: NextRequest, event: NextFetchEvent) {
pug.track(request, undefined, event)
return NextResponse.next()
}The proxy runs before the page, so these reports carry no status code.
Express and node:http
import { crawlers } from '@pug-sh/crawlers'
app.use(crawlers({ apiKey: process.env.PUG_PRIVATE_KEY }).express())The middleware reports after the response finishes. With plain node:http, call
it at the start of your handler: middleware(req, res).
Any other server
Report a request yourself with one POST, from any language:
curl https://api.pugs.dev/sdk.events.v1.EventsService/BatchCreate \
-H 'x-api-key: prv_...' \
-H 'content-type: application/json' \
-d '{"events": [{"eventId": "0193b2c4-8f1e-7a2b-9c3d-4e5f6a7b8c9d", "occurTime": "2026-11-14T09:30:00Z",
"kind": "crawler_request",
"client": {"userAgent": "... GPTBot/1.2 ...", "ip": "20.171.207.5"},
"autoProperties": {"$url": {"stringValue": "https://example.com/pricing"},
"$status_code": {"intValue": 200}}}]}'One call takes up to 1,000 requests. Make eventId (a UUID) once and resend it
unchanged on a retry, so Pug stores the request once.
Options
| Option | Default | What it does |
|---|---|---|
apiKey |
required | A private key, prv_.... |
endpoint |
https://api.pugs.dev |
Where Pug runs. A self-hosted install sets its own. |
categories |
all five | Which kinds of crawler to report: answers, search, training, seo, other. |
skip |
none | Agents to leave out, by name, such as ['Googlebot']. |
includeAssets |
false |
Also report requests for scripts, styles, images and fonts. |
getIP |
the platform’s | Returns the crawler’s address for a request. |
The categories
| Category | Meaning | Examples |
|---|---|---|
answers |
Fetched live because a person asked an AI product | ChatGPT-User, Claude-User, Perplexity-User |
search |
Builds a search index that results and AI answers draw on | OAI-SearchBot, Googlebot, bingbot |
training |
Collects pages to train models | GPTBot, ClaudeBot, CCBot |
seo |
Crawls for an SEO tool | AhrefsBot, SemrushBot, MJ12bot |
other |
AI-related, but none of the above | OAI-AdsBot, GoogleOther |
The AI agents come from Pug’s own table of 35, each with its company and, where the operator publishes them, its IP ranges. Search engines and SEO tools come from the crawler-user-agents list. Both ship inside the package, so a new crawler reaches your site with the package’s next release.
What it sends
Only requests whose user agent names a known crawler, so your visitors’ requests never leave your server. For each one it sends:
| Field | Value |
|---|---|
$url |
The full URL, without its query string or fragment |
client.userAgent |
The User-Agent header |
client.ip |
The client’s address, for the IP check. Pug never stores it. |
$status_code |
The response’s status, when the adapter sees it |
It never sends query strings, cookies, other headers, bodies, requests other
than GET and HEAD, or (unless includeAssets is on) static files.
Delivery
One POST per crawler request, after the response, with a 2-second timeout and one retry on a network error or a server error. It never throws, so it can’t break your site, and it logs a failure once per process.
The client’s address
The package reads CF-Connecting-IP on Cloudflare, x-real-ip on Vercel,
req.ip on Express (which follows its trust proxy setting) and the socket on
plain Node. Pass getIP when that’s wrong:
- Behind a CDN, your server sees the CDN’s address, so every request fails
the IP check. Read the CDN’s client-IP header in
getIP. - A fetch-style server that Cloudflare or Vercel doesn’t front can’t trust
those two headers, since anyone can send them. Return the socket’s address,
such as Bun’s
server.requestIP(req)?.address.
Running at the edge (Workers, Vercel) sees every request, including those a CDN would answer from its cache.
See them in Pug
Each request is a crawler_request event, tagged as a bot, so it stays out of
your other numbers. Insights and the Events list include bots on their own when
you ask for crawler_request or one of its properties:
| Property | Value |
|---|---|
$bot_reason |
The agent’s name, such as GPTBot |
$crawler_company |
Such as OpenAI; none for a search engine or SEO tool from the list |
$crawler_category |
answers, search, training, seo or other |
$crawler_verified |
verified (inside the operator’s ranges), failed (outside them, likely an impostor) or unchecked |
$status_code |
The response’s status, when sent |
$url, $pathname, $hostname |
The page requested |
Break crawler_request down by $crawler_category for requests over time, rank
$crawler_company or $bot_reason with Top K, or filter $status_code to 404
for the pages crawlers can’t find. A crawler is never listed as a person.
Volume
Crawler requests count toward your Pug usage like any event, and a busy site
can get more of them than page views. SEO tools are among the busiest crawlers,
so seo is the first category to leave out.
Self-hosted Pug
Point endpoint at your Pug. Its server fetches each operator’s IP ranges when
it starts, then every 6 hours. Without internet
access, set PUG_CRAWLER_RANGES_ENABLED=false, and every request reads
unchecked.