host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://spotlights-feed.github.com/robots.txt 404 2795 bytes 0 User-agent groups
Raw robots.txt 
<!doctype html>
<html>
  <head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta http-equiv="content-language" content="en-US">
  <meta name="robots" content="noindex">
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">

  <title></title>

  
    <link rel="icon" type="image/x-icon" href="https://spotlights-feed.github.com/favicon.ico">
  

  <link rel="stylesheet" type="text/css" href="https://spotlights-feed.github.com/assets/styles.css">
</head>

  <body>
    <header class="border-bottom bg-gray-light">
  <div class="container-xl p-responsive py-3">
    <div class="d-sm-flex flex-items-center gutter-sm">
      <h1 class="f4 text-normal mx-sm-3 mb-3 mb-sm-0"><a href="https://spotlights-feed.github.com" class="d-flex flex-items-center link-gray-dark no-underline"><svg height="32" class="octicon octicon-mark-github mr-2" aria-label="github-invertocat" viewBox="0 0 16 16" version="1.1" width="32" role="img"><path fill-rule="evenodd" d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0016 8c0-4.42-3.58-8-8-8z"></path></svg><span class="d-none">GitHub</span> Spotlights</a></h1>

      
        <a href="https://spotlights-feed.github.com/spotlights.json" class="btn btn-outline mx-sm-3">JSON</a>
      
    </div>
  </div>
</header>


    
    <div class="container-xl p-responsive">
      <h1 class="alt-h2 border-bottom pb-3 mt-5 mb-6">/404.html</h1>

      <div class="col-lg-5">
        <article class="my-3 border rounded-1 box-shadow bg-white">
  

  <div class="border-top bg-white">
    <div class="Spotlight-text p-3">
      
      <h1 class="h3-mktg text-gray-dark lh-condensed mb-2">
        
        <a href="">
          
            
          
        </a>
      </h1>
      <div class="markdown-body f5 text-gray">
        <style type="text/css" media="screen">
  .container {
    margin: 10px auto;
    max-width: 600px;
    text-align: center;
  }
  h1 {
    margin: 30px 0;
    font-size: 4em;
    line-height: 1;
    letter-spacing: -1px;
  }
</style>

<div class="container">
  <h1>404</h1>

  <p><strong>Page not found :(</strong></p>
  <p>The requested page could not be found.</p>
</div>

      </div>
    </div>
  </div>
</article>

      </div>
    </div>
  </body>
</html>
Sponsored host.tools Pro
Bulk lookups · monitors · webhooks · 10,000 API calls/day

Upgrade to Pro for $19/mo. Cancel anytime. Works with the same API you already use.
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fspotlights-feed.github.com 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fspotlights-feed.github.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view