---
title: "See the AI and search crawlers reading your site"
description: "The Overview shows which AI products, search engines and SEO tools read your site, reported by your own server through @pug-sh/crawlers."
date: 2026-11-17
tags: ["Dashboard","Crawlers"]
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.pug.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# See the AI and search crawlers reading your site

Crawlers fetch HTML and never run JavaScript, so a browser script can't see
them. [`@pug-sh/crawlers`](/crawlers/) runs in your site's own request handling
instead and reports each crawler request to Pug.

## New

- The Overview has an **AI and search crawlers** section: requests by category,
  the crawlers, companies and categories behind them, whether each was
  verified, and the pages they read, 404s included.
- Events names the crawler on its rows, and a crawler's id opens a page of its
  own.
- Dashboards offer two crawler tile templates.
- Insights and the tile editor have a bots chip. A query that reads crawler
  data includes bot traffic on its own, and the chip shows it locked.

Source: https://docs.pug.sh/changelog/2026-11-17-ai-crawlers/index.mdx
