Skip to content
OMBZ
Sign in

OmbzBot

OmbzBot is the web crawler of OMBZ, a tool that shows website owners how their websites perform in search. It checks pages for technical issues such as broken links, redirects and missing titles.

Which websites it visits

OmbzBot only crawls websites whose owners added them to OMBZ. It does not crawl the web at large. Speed measurements (Core Web Vitals) are made by Google PageSpeed Insights, not by OmbzBot.

How to recognise it

OmbzBot identifies itself with this user agent:

OmbzBot/1.0 (+https://ombz.com/bot)

How it behaves

  • It reads robots.txt for every host it visits and follows the rules for OmbzBot, or the rules for all crawlers (*) when there is no OmbzBot group.
  • When robots.txt does not exist (a 4xx answer), it crawls the whole website. When robots.txt answers with a server error or cannot be reached, it does not crawl that website at all.
  • It respects Crawl-delay, up to 60 seconds between requests.
  • It makes at most 2 requests at the same time to a website, and only 1 when a Crawl-delay is set.
  • It waits at least 500 ms between requests to the same host, over http and https together.
  • When a website answers 429 Too Many Requests, it waits as long as the Retry-After header asks, up to 60 seconds, and fetches the page again later. After 10 such answers in a row it stops the crawl.
  • It does not follow page redirects automatically: each redirect target is a separate request that obeys robots.txt.
  • It fetches at most a set number of pages per crawl, 2,000 by default.
  • Scheduled crawls run weekly or monthly, at night (Europe/Amsterdam). The owner of a website can also start a crawl by hand.
  • It only analyses HTML pages and reads at most 5 MB of each page.
  • It only connects to public addresses.

How to slow it down

To make OmbzBot wait longer between requests, set a Crawl-delay in seconds for it in your robots.txt:

User-agent: OmbzBot
Crawl-delay: 10

How to block it

To keep OmbzBot away from your whole website, add this to your robots.txt:

User-agent: OmbzBot
Disallow: /

Contact

Questions or problems with OmbzBot? Email us at privacy@ombz.com.