bubblio

BubblioBot

BubblioBot is the crawler behind Bubblio's knowledge feature: a website owner adds their own site in the Bubblio dashboard, BubblioBot reads its public pages, and that content grounds their agent door — when visiting AI agents ask questions, the ask tool answers from it, with citations. It identifies itself on every request as:

BubblioBot/1.0 (+https://bubblio.dev/bot)

What it does — and doesn't

  • Opt-in only. It crawls a site only after that site's owner explicitly adds the URL in their Bubblio dashboard. It never discovers or wanders onto third-party sites — crawls stay on the same site, including through redirects.
  • Public pages only. Plain GET requests, no cookies, no logins, no form submissions — it can only see what any logged-out visitor (or search engine) sees.
  • robots.txt first. It fetches and honors robots.txt before anything else (matching User-agent: BubblioBot, else *), and skips pages marked noindex. If robots.txt can't be read, it doesn't crawl.
  • Polite. About one request per second, sitemap-first, roughly 200 pages per site, 2 MB per page, and a re-visit about once a week (or when the owner requests a refresh).

What happens to the content

  • Indexed content powers only the site owner's own agent door — it is never shared across Bubblio customers and never used to train AI models.
  • When the owner removes a source (or their account), its indexed content is deleted immediately.
  • Pages that disappear from a site, or that add noindex, are dropped from the index on the next crawl.

Blocking or limiting BubblioBot

Add a rule to your site's robots.txt — it takes effect on the next crawl:

# Block BubblioBot entirely
User-agent: BubblioBot
Disallow: /

# …or keep it out of specific paths only
User-agent: BubblioBot
Disallow: /private/

Questions

Seeing unexpected traffic from this user-agent, or have a question about how your content is handled? Email bubblio@andresio.com and include your domain — we'll respond quickly.