Our crawler
Our crawler identifies itself with this user agent:
SailboatsBot/0.1 (+https://github.com/karooolis/sailboats-app)
What it does
- It only visits sites whose terms and robots.txt don't forbid it, and it always follows robots.txt.
- It reads listing index pages every 6 hours, one page at a time, at least 3 seconds apart.
- It never logs in, fills in forms or gets around bot protection.
- It keeps only facts (model, year, length, price, location, link), never ad texts or photos.
How to block it
Add this to your site's robots.txt; the crawler stops on its next visit:
User-agent: SailboatsBot Disallow: /
To also remove what's already listed here, see opt-out and takedown.