robots.txt parser

1. 1
   Paste your input

   Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

2. 2
   Click "Inspect"

   host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

3. 3
   Open the API tab

   Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.