Skip to content
pugdocs

See the AI and search crawlers reading your site

The Overview shows which AI products, search engines and SEO tools read your site, reported by your own server through @pug-sh/crawlers.

Dashboard, Crawlers

Crawlers fetch HTML and never run JavaScript, so a browser script can’t see them. @pug-sh/crawlers runs in your site’s own request handling instead and reports each crawler request to Pug.

New

  • The Overview has an AI and search crawlers section: requests by category, the crawlers, companies and categories behind them, whether each was verified, and the pages they read, 404s included.
  • Events names the crawler on its rows, and a crawler’s id opens a page of its own.
  • Dashboards offer two crawler tile templates.
  • Insights and the tile editor have a bots chip. A query that reads crawler data includes bot traffic on its own, and the chip shows it locked.

Type to search...

↑↓ navigate↵ selectEsc close