Crawler
Every request we make points here
- Sites tracked
- 339
- Prices recorded
- 13,362
How to recognise us
Our user-agent string is exactly:
creatoradviseBot/1.0 (+https://creatoradvise.com/crawler)We do not disguise our crawler as a browser, and we do not rotate user-agents. If a request claims to be creatoradviseBot but does not come from this identity, it is not us.
What we fetch and how often
We build a price comparison of social media growth providers. To do that we read the pages you already publish publicly: your services or pricing page, and your homepage for a reachability check.
- Prices — weekly. Measured against our own captures, 92% of prices were unchanged after nine hours and the 8% that moved shifted by 3.6% on average. Weekly is enough to catch real movement, and it halves the load we put on you.
- Reachability — hourly. A single request to your homepage, reading at most 50KB. This is how we tell whether a provider is still online.
- Rate limiting. At least ten seconds between requests to the same host, randomised upward. We do not run parallel requests against one host.
- robots.txt. Checked and obeyed before every fetch, including for browser-rendered pages.
We do not create accounts, we do not submit forms, we do not attempt to access anything behind a login, and we do not place orders.
How to limit or block us
Add a rule to your robots.txt and we will honour it on the next fetch:
User-agent: creatoradviseBot
Disallow: /You can also email us at support@creatoradvise.com and we will stop, and remove your listing if you want it gone. We will not argue with you about it.
If you would rather correct what we show than remove it, you can claim your profile by proving domain ownership.
