What it is

GoldilocksBot is the crawler behind goldilocks.fit. Once a day it reads the public catalogue pages of the makers we cover and records what changed: new sculpts, sizes and pours, published measurements, prices, and stock. That's what keeps our specs current and the drops page honest.

It identifies itself on every request:

GoldilocksBot/0.1 (+https://goldilocks.fit/bot)

How it behaves

• robots.txt is law. The bot fetches and obeys each site's robots.txt, including crawl-delay, before touching anything else. Disallowed paths are never fetched.
• Rate-limited. Requests are spaced out per host — a daily catalogue pass, not a scrape-storm.
• Public pages only. It reads what any browser sees: product pages and public catalogue endpoints. No accounts, no forms, no checkout flows.
• Honest caching. Pages are fetched once per run and stored, so we never hammer the same URL twice in a day.

If you run one of these shops

We want this to be a good deal for you: accurate specs and live stock on our side send buyers to your site, not away from it. If you'd rather tune how we crawl — or not be crawled at all — either add GoldilocksBot to your robots.txt (it will comply on the next run) or email us and we'll sort it by hand.

Something looks wrong?

If you're seeing unexpected traffic claiming to be GoldilocksBot, or the bot is misbehaving in any way, please get in touch with the timestamps and we'll investigate — impersonation included.