If you found this address in your server logs, it was left by our crawler. It fetches pages so that a UniversalChatbot customer can answer questions from them, and it only ever visits addresses that customer gave us.
User agentUniversalChatbot/1.0 (+https://universalchatbot.com/bot)
Why it visited your site.
Every crawl is started by a customer, for their own knowledge base. We do not crawl the open web, and we do not build a general index.
Who asked for it
A UniversalChatbot customer added your address to their knowledge base. That is the only way a page enters the queue. If the address was yours to give, the crawl is one you asked for.
What it reads
The text of the pages it is pointed at, plus the links on them so it can follow a site section, and ordinary page metadata such as the title and description.
What it does with it
The text is converted to plain markdown, split into passages and indexed for search, so the customer's flows and drafted replies can quote from it. It is stored against that customer's account and nobody else's.
What it never sees
It signs in to nothing. It sends no credentials, fills in no forms and follows no link that needs a session, so anything behind a login stays where it is.
How it behaves.
A crawler is a guest on your infrastructure. These are the limits it runs under.
It identifies itself
Every request carries the user agent above, with a link back to this page, so it is never anonymous in your logs.
A few requests at a time
It holds a small number of requests open at once and waits for them to finish before opening more, so a crawl arrives as a trickle rather than a burst.
It stays on one host
It follows links only within the site it was pointed at, and stops at a page limit the customer sets, so a crawl has a known size before it starts.
If you would rather it did not.
The crawler does not currently read robots.txt, so blocking it there will not stop it. Two things will.
Block the user agent
Refuse requests carrying UniversalChatbot/1.0 at your web server, CDN or firewall. This takes effect immediately and needs nothing from us.
Write to us
Send us the address and we will stop it being fetched again, and take it up with the customer who queued it.