host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://www.vanhope.ai/robots.txt 200 2623 bytes 0 User-agent groups
Raw robots.txt 
<!doctype html>
<html lang="zh-CN">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/png" href="/logo.png" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vanhope - AI API Gateway</title>
    <script type="module" crossorigin src="/assets/index-C_n0JMsQ.js"></script>
    <link rel="modulepreload" crossorigin href="/assets/vendor-vue-C1PG35FK.js">
    <link rel="modulepreload" crossorigin href="/assets/vendor-i18n-CS7xxe6p.js">
    <link rel="modulepreload" crossorigin href="/assets/vendor-misc-BVDJZfTg.js">
    <link rel="stylesheet" crossorigin href="/assets/vendor-misc-DB0Q8XAf.css">
    <link rel="stylesheet" crossorigin href="/assets/index-hMTsfnYR.css">
  <script nonce="4FVXyQJnA+qbgq9IH7uYuQ==">window.__APP_CONFIG__={"registration_enabled":false,"email_verify_enabled":false,"registration_email_suffix_whitelist":[],"promo_code_enabled":true,"password_reset_enabled":false,"invitation_code_enabled":false,"totp_enabled":false,"login_agreement_enabled":false,"login_agreement_mode":"modal","login_agreement_updated_at":"2026-03-31","login_agreement_revision":"0eff189d66e0af4e","login_agreement_documents":[{"id":"terms","title":"服务条款","content_md":""},{"id":"usage-policy","title":"使用政策","content_md":""},{"id":"supported-regions","title":"支持的国家和地区","content_md":""},{"id":"service-specific-terms","title":"服务特定条款","content_md":""}],"turnstile_enabled":false,"turnstile_site_key":"","site_name":"Vanhope","site_logo":"/logo.png","site_subtitle":"-","api_base_url":"https://vanhope.ai","contact_info":"","doc_url":"","home_content":"https://vanhope.ai/home.html","hide_ccs_import_button":false,"purchase_subscription_enabled":false,"purchase_subscription_url":"","table_default_page_size":20,"table_page_size_options":[10,20,50],"custom_menu_items":[],"custom_endpoints":[],"linuxdo_oauth_enabled":false,"wechat_oauth_enabled":false,"wechat_oauth_open_enabled":false,"wechat_oauth_mp_enabled":false,"wechat_oauth_mobile_enabled":false,"oidc_oauth_enabled":false,"oidc_oauth_provider_name":"OIDC","github_oauth_enabled":false,"google_oauth_enabled":false,"backend_mode_enabled":false,"payment_enabled":false,"version":"","balance_low_notify_enabled":false,"account_quota_notify_enabled":false,"balance_low_notify_threshold":0,"balance_low_notify_recharge_url":"","channel_monitor_enabled":true,"channel_monitor_default_interval_seconds":60,"available_channels_enabled":false,"affiliate_enabled":false,"risk_control_enabled":false};</script></head>
  <body>
    <div id="app"></div>
  </body>
</html>
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fwww.vanhope.ai 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fwww.vanhope.ai'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view