---
title: "AI crawlers"
description: "See which AI products, search engines and SEO tools read your site, and why, with @pug-sh/crawlers in your own request handling and a private key."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.pug.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# AI crawlers

See which AI products, search engines and SEO tools read your site, and why: to
answer someone's question, to build a search index, or to train a model.

Crawlers fetch HTML and don't run JavaScript, so a browser analytics script never
sees them. This package runs in your site's own request handling instead. When a
request comes from a known crawler, it sends one small report to
Pug after the response. Pug checks the request's IP against the
ranges the crawler's operator publishes, so a scraper borrowing GPTBot's name
reads as unverified.

It has no dependencies and runs on Cloudflare Workers, Vercel, Next.js, Hono,
Bun, Deno, Express and plain Node.

## Install

```sh
npm install @pug-sh/crawlers
```

You need a private key (`prv_...`) from your project's settings in Pug. Keep it
on the server, in an environment variable: it can also read your analytics.

## Cloudflare Workers and Pages Functions

```ts
import { crawlers } from '@pug-sh/crawlers'

export default {
  async fetch(request, env, ctx) {
    const response = await env.ASSETS.fetch(request)
    crawlers({ apiKey: env.PUG_PRIVATE_KEY }).track(request, response, ctx)
    return response
  },
}
```

Passing `ctx` lets the Worker finish the report after the response is sent.

## Hono, Bun and Deno

```ts
import { crawlers } from '@pug-sh/crawlers'

const pug = crawlers({ apiKey: process.env.PUG_PRIVATE_KEY })

app.use(async (c, next) => {
  await next()
  pug.track(c.req.raw, c.res)
})
```

On Cloudflare, pass `c.executionCtx` as the third argument. Elsewhere, see
[the client's address](#the-clients-address).

## Next.js and Vercel

```ts
// proxy.ts
import { crawlers } from '@pug-sh/crawlers'
import { type NextFetchEvent, type NextRequest, NextResponse } from 'next/server'

const pug = crawlers({ apiKey: process.env.PUG_PRIVATE_KEY! })

export function proxy(request: NextRequest, event: NextFetchEvent) {
  pug.track(request, undefined, event)
  return NextResponse.next()
}
```

The proxy runs before the page, so these reports carry no status code.

## Express and node:http

```ts
import { crawlers } from '@pug-sh/crawlers'

app.use(crawlers({ apiKey: process.env.PUG_PRIVATE_KEY }).express())
```

The middleware reports after the response finishes. With plain `node:http`, call
it at the start of your handler: `middleware(req, res)`.

## Any other server

Report a request yourself with one POST, from any language:

```sh
curl https://api.pugs.dev/sdk.events.v1.EventsService/BatchCreate \
  -H 'x-api-key: prv_...' \
  -H 'content-type: application/json' \
  -d '{"events": [{"eventId": "0193b2c4-8f1e-7a2b-9c3d-4e5f6a7b8c9d", "occurTime": "2026-11-14T09:30:00Z",
       "kind": "crawler_request",
       "client": {"userAgent": "... GPTBot/1.2 ...", "ip": "20.171.207.5"},
       "autoProperties": {"$url": {"stringValue": "https://example.com/pricing"},
                          "$status_code": {"intValue": 200}}}]}'
```

One call takes up to 1,000 requests. Make `eventId` (a UUID) once and resend it
unchanged on a retry, so Pug stores the request once.

## Options

| Option          | Default                | What it does                                                                       |
| --------------- | ---------------------- | ---------------------------------------------------------------------------------- |
| `apiKey`        | required               | A private key, `prv_...`.                                                          |
| `endpoint`      | `https://api.pugs.dev` | Where Pug runs. A self-hosted install sets its own.                                |
| `categories`    | all five               | Which kinds of crawler to report: `answers`, `search`, `training`, `seo`, `other`. |
| `skip`          | none                   | Agents to leave out, by name, such as `['Googlebot']`.                             |
| `includeAssets` | `false`                | Also report requests for scripts, styles, images and fonts.                        |
| `getIP`         | the platform's         | Returns the crawler's address for a request.                                       |

## The categories

| Category   | Meaning                                                   | Examples                                   |
| ---------- | --------------------------------------------------------- | ------------------------------------------ |
| `answers`  | Fetched live because a person asked an AI product         | ChatGPT-User, Claude-User, Perplexity-User |
| `search`   | Builds a search index that results and AI answers draw on | OAI-SearchBot, Googlebot, bingbot          |
| `training` | Collects pages to train models                            | GPTBot, ClaudeBot, CCBot                   |
| `seo`      | Crawls for an SEO tool                                    | AhrefsBot, SemrushBot, MJ12bot             |
| `other`    | AI-related, but none of the above                         | OAI-AdsBot, GoogleOther                    |

The AI agents come from Pug's own table of 35, each with its company and, where
the operator publishes them, its IP ranges. Search engines and SEO tools come
from the [crawler-user-agents](https://github.com/monperrus/crawler-user-agents)
list. Both ship inside the package, so a new crawler reaches your site with the
package's next release.

## What it sends

Only requests whose user agent names a known crawler, so your visitors' requests
never leave your server. For each one it sends:

| Field              | Value                                                        |
| ------------------ | ------------------------------------------------------------ |
| `$url`             | The full URL, without its query string or fragment           |
| `client.userAgent` | The User-Agent header                                        |
| `client.ip`        | The client's address, for the IP check. Pug never stores it. |
| `$status_code`     | The response's status, when the adapter sees it              |

It never sends query strings, cookies, other headers, bodies, requests other
than GET and HEAD, or (unless `includeAssets` is on) static files.

## Delivery

One POST per crawler request, after the response, with a 2-second timeout and
one retry on a network error or a server error. It never throws, so it can't
break your site, and it logs a failure once per process.

## The client's address

The package reads `CF-Connecting-IP` on Cloudflare, `x-real-ip` on Vercel,
`req.ip` on Express (which follows its `trust proxy` setting) and the socket on
plain Node. Pass `getIP` when that's wrong:

- **Behind a CDN**, your server sees the CDN's address, so every request fails
  the IP check. Read the CDN's client-IP header in `getIP`.
- **A fetch-style server that Cloudflare or Vercel doesn't front** can't trust
  those two headers, since anyone can send them. Return the socket's address,
  such as Bun's `server.requestIP(req)?.address`.

Running at the edge (Workers, Vercel) sees every request, including those a CDN
would answer from its cache.

## See them in Pug

Each request is a `crawler_request` event, tagged as a bot, so it stays out of
your other numbers. Insights and the Events list include bots on their own when
you ask for `crawler_request` or one of its properties:

| Property | Value |
|---|---|
| `$bot_reason` | The agent's name, such as `GPTBot` |
| `$crawler_company` | Such as `OpenAI`; none for a search engine or SEO tool from the list |
| `$crawler_category` | `answers`, `search`, `training`, `seo` or `other` |
| `$crawler_verified` | `verified` (inside the operator's ranges), `failed` (outside them, likely an impostor) or `unchecked` |
| `$status_code` | The response's status, when sent |
| `$url`, `$pathname`, `$hostname` | The page requested |

Break `crawler_request` down by `$crawler_category` for requests over time, rank
`$crawler_company` or `$bot_reason` with Top K, or filter `$status_code` to 404
for the pages crawlers can't find. A crawler is never listed as a person.

## Volume

Crawler requests count toward your Pug usage like any event, and a busy site
can get more of them than page views. SEO tools are among the busiest crawlers,
so `seo` is the first category to leave out.

## Self-hosted Pug

Point `endpoint` at your Pug. Its server fetches each operator's IP ranges when
it starts, then every 6 hours. Without internet
access, set `PUG_CRAWLER_RANGES_ENABLED=false`, and every request reads
`unchecked`.

Source: https://docs.pug.sh/crawlers/index.mdx
