About StackProspectBot

If you found this page in your server logs: StackProspectBot is the crawler behind StackProspect. It looks at the public front of company websites to record which infrastructure and software vendors they use. It identifies itself honestly and is built to be a polite guest.

What it does

• Fetches robots.txt first and obeys it. Disallowed paths are never requested.
• Requests the homepage and a handful of linked public pages and scripts, at most one request at a time per host with a two-second pause between them.
• Backs off on 429 responses and stops after repeated 403s. It never retries around blocks or rotates identities.
• Reads only what any browser would receive. It never logs in, never submits forms, never guesses hidden URLs, never tests for vulnerabilities.
• Records vendor evidence such as response headers, DNS records, cookie names and script hosts. It never stores credentials, session identifiers or personal data.
• Revisits a site between daily and monthly depending on how often it changes.

Identification

User-Agent: StackProspectBot/1.0 (+https://stackprospect.com/scanner). To block it entirely, add to your robots.txt:

User-agent: StackProspectBot
Disallow: /

Opt out

Prefer your domain excluded altogether? Email [email protected] with the domain and it goes on the suppression list, normally within one business day. Suppressed domains and all their subdomains are never fetched again.