crawloops bot
This crawler identifies itself in its User-Agent and honors robots.txt and per-host rate limits.
To adjust or exclude crawling of your site, use robots.txt
(user-agent: crawloopsbot) or contact us directly. We
fetch only publicly served pages.
- User-Agent token
crawloopsbot- Robots.txt directive
User-agent: crawloopsbot- Contact
- hello@crawloops.com