host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://support.enterprise.github.com/robots.txt 200 2454 bytes 2 User-agent groups
User-agent: *
Allow
  • /hc/*/requests/new
Disallow
  • /children
  • /groups
  • /organizations
  • /requests
  • /registration
  • /plans
  • /accounts
  • /account
  • /proxy
  • /rules
  • /tags
  • /ticket_fields
  • /reports
  • /search
  • /slas
  • /integrations
  • /users
  • /suspended_tickets
  • /events
  • /console
  • /requests/*/satisfaction/
  • /hc/activity
  • /hc/change_language/
  • /hc/communities/public/topics/*?*filter=
  • /hc/communities/public/questions$
  • /hc/communities/public/questions?*filter=
  • /hc/communities/public/questions/unanswered
  • /hc/*/signin
  • /hc/requests/
  • /hc/*/requests/
  • /hc/*/search
  • /api/v2/help_center/*/articles/*/stats/view
  • /api/v2/help_center/community/posts/*/stats/view
  • /access/normal
  • /access/sso_bypass
  • /access/unauthenticated
  • /theming
  • /knowledge
  • /access/
  • /auth/
  • /cdn-cgi/
  • /tickets
User-agent: *
Allow
  • /hc/*/requests/new
Disallow
  • /children
  • /groups
  • /organizations
  • /requests
  • /registration
  • /plans
  • /accounts
  • /account
  • /proxy
  • /rules
  • /tags
  • /ticket_fields
  • /reports
  • /search
  • /slas
  • /integrations
  • /users
  • /suspended_tickets
  • /events
  • /console
  • /requests/*/satisfaction/
  • /hc/activity
  • /hc/change_language/
  • /hc/communities/public/topics/*?*filter=
  • /hc/communities/public/questions$
  • /hc/communities/public/questions?*filter=
  • /hc/communities/public/questions/unanswered
  • /hc/*/signin
  • /hc/requests/
  • /hc/*/requests/
  • /hc/*/search
  • /api/v2/help_center/*/articles/*/stats/view
  • /api/v2/help_center/community/posts/*/stats/view
  • /access/normal
  • /access/sso_bypass
  • /access/unauthenticated
  • /theming
  • /knowledge
  • /access/
  • /auth/
  • /cdn-cgi/
  • /tickets
Raw robots.txt 
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file

User-agent: AppleBot # Allow /tickets
Disallow: /children
Disallow: /groups
Disallow: /organizations
Disallow: /requests
Disallow: /registration
Disallow: /plans
Disallow: /accounts
Disallow: /account
Disallow: /proxy
Disallow: /rules
Disallow: /tags
Disallow: /ticket_fields
Disallow: /reports
Disallow: /search
Disallow: /slas
Disallow: /integrations
Disallow: /users
Disallow: /suspended_tickets
Disallow: /events
Disallow: /console
Disallow: /requests/*/satisfaction/
Disallow: /hc/activity
Disallow: /hc/change_language/
Disallow: /hc/communities/public/topics/*?*filter=
Disallow: /hc/communities/public/questions$
Disallow: /hc/communities/public/questions?*filter=
Disallow: /hc/communities/public/questions/unanswered
Disallow: /hc/*/signin
Disallow: /hc/requests/
Disallow: /hc/*/requests/
Allow: /hc/*/requests/new
Disallow: /hc/*/search
Disallow: /api/v2/help_center/*/articles/*/stats/view
Disallow: /api/v2/help_center/community/posts/*/stats/view
Disallow: /access/normal
Disallow: /access/sso_bypass
Disallow: /access/unauthenticated
Disallow: /theming
Disallow: /knowledge
Disallow: /access/
Disallow: /auth/

Disallow: /cdn-cgi/

User-agent: *
Disallow: /children
Disallow: /groups
Disallow: /organizations
Disallow: /requests
Disallow: /registration
Disallow: /plans
Disallow: /accounts
Disallow: /account
Disallow: /proxy
Disallow: /rules
Disallow: /tags
Disallow: /ticket_fields
Disallow: /reports
Disallow: /search
Disallow: /slas
Disallow: /integrations
Disallow: /users
Disallow: /suspended_tickets
Disallow: /events
Disallow: /console
Disallow: /requests/*/satisfaction/
Disallow: /hc/activity
Disallow: /hc/change_language/
Disallow: /hc/communities/public/topics/*?*filter=
Disallow: /hc/communities/public/questions$
Disallow: /hc/communities/public/questions?*filter=
Disallow: /hc/communities/public/questions/unanswered
Disallow: /hc/*/signin
Disallow: /hc/requests/
Disallow: /hc/*/requests/
Allow: /hc/*/requests/new
Disallow: /hc/*/search
Disallow: /api/v2/help_center/*/articles/*/stats/view
Disallow: /api/v2/help_center/community/posts/*/stats/view
Disallow: /access/normal
Disallow: /access/sso_bypass
Disallow: /access/unauthenticated
Disallow: /theming
Disallow: /knowledge
Disallow: /access/
Disallow: /auth/

Disallow: /cdn-cgi/
Disallow: /tickets


Sitemap: https://enterprise.githubsupport.com/hc/sitemap.xml
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fsupport.enterprise.github.com 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fsupport.enterprise.github.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view