host.tools

robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools
Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector
https://home.single10.app/robots.txt 200 8375 bytes 0 User-agent groups
Raw robots.txt 



<!DOCTYPE html>

<html>
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
        
        <meta name="darkreader-lock">
        <title>Single10</title>
        <link rel="icon" href="/static/dist/assets/icons/icon.png">
        <link rel="shortcut icon" href="/static/dist/assets/icons/icon.png">
        

        
<link rel="prefetch" href="/static/dist/assets/images/flow_background.jpg" />




<script>
    window.authentik = {
        locale: "en",
        config: JSON.parse('{\u0022error_reporting\u0022: {\u0022enabled\u0022: false, \u0022sentry_dsn\u0022: \u0022https://[email protected]\u002Dreporting.a7k.io/4504163677503489\u0022, \u0022environment\u0022: \u0022customer\u0022, \u0022send_pii\u0022: false, \u0022traces_sample_rate\u0022: 0.1}, \u0022capabilities\u0022: [\u0022can_save_media\u0022, \u0022can_asn\u0022, \u0022can_geo_ip\u0022, \u0022can_impersonate\u0022, \u0022is_enterprise\u0022], \u0022cache_timeout\u0022: 300, \u0022cache_timeout_flows\u0022: 300, \u0022cache_timeout_policies\u0022: 300}'),
        brand: JSON.parse('{\u0022matched_domain\u0022: \u0022authentik\u002Ddefault\u0022, \u0022branding_title\u0022: \u0022Single10\u0022, \u0022branding_logo\u0022: \u0022/static/dist/assets/icons/icon_left_brand.svg\u0022, \u0022branding_favicon\u0022: \u0022/static/dist/assets/icons/icon.png\u0022, \u0022branding_custom_css\u0022: \u0022/* Move SSO section above form */\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter {\u005Cn    order: \u002D1 !important\u003B\u005Cn    padding\u002Dbottom: 16px !important\u003B\u005Cn    margin\u002Dbottom: 16px !important\u003B\u005Cn    border\u002Dbottom: 1px solid rgba(255,255,255,0.1) !important\u003B\u005Cn}\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dbody {\u005Cn    display: flex !important\u003B\u005Cn    flex\u002Ddirection: column !important\u003B\u005Cn}\u005Cn\u005Cn/* Make SSO button prominent */\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter\u002Dlinks\u002Ditem a {\u005Cn    display: flex !important\u003B\u005Cn    align\u002Ditems: center !important\u003B\u005Cn    justify\u002Dcontent: center !important\u003B\u005Cn    gap: 12px !important\u003B\u005Cn    width: 100% !important\u003B\u005Cn    padding: 14px 24px !important\u003B\u005Cn    background: #4285f4 !important\u003B\u005Cn    color: #fff !important\u003B\u005Cn    border\u002Dradius: 8px !important\u003B\u005Cn    font\u002Dsize: 16px !important\u003B\u005Cn    font\u002Dweight: 600 !important\u003B\u005Cn    text\u002Ddecoration: none !important\u003B\u005Cn    transition: background 0.15s !important\u003B\u005Cn    min\u002Dheight: 48px !important\u003B\u005Cn}\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter\u002Dlinks\u002Ditem a:hover {\u005Cn    background: #3367d6 !important\u003B\u005Cn}\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter\u002Dlinks\u002Ditem a img {\u005Cn    width: 24px !important\u003B\u005Cn    height: 24px !important\u003B\u005Cn}\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter\u002Dlinks {\u005Cn    display: flex !important\u003B\u005Cn    flex\u002Ddirection: column !important\u003B\u005Cn    padding: 0 !important\u003B\u005Cn    margin: 0 !important\u003B\u005Cn    gap: 8px !important\u003B\u005Cn}\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter\u002Dlinks\u002Ditem {\u005Cn    width: 100% !important\u003B\u005Cn}\u005Cn\u005Cn/* Divider */\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dfooter::after {\u005Cn    content: \u005C\u0022or sign in with email\u005C\u0022 !important\u003B\u005Cn    display: block !important\u003B\u005Cn    text\u002Dalign: center !important\u003B\u005Cn    font\u002Dsize: 13px !important\u003B\u005Cn    color: rgba(255,255,255,0.4) !important\u003B\u005Cn    margin\u002Dtop: 16px !important\u003B\u005Cn}\u005Cn\u005Cn/* De\u002Demphasize the password form */\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dbody .pf\u002Dc\u002Dform {\u005Cn    opacity: 0.5 !important\u003B\u005Cn    transition: opacity 0.2s !important\u003B\u005Cn}\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dbody .pf\u002Dc\u002Dform:hover,\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dbody .pf\u002Dc\u002Dform:focus\u002Dwithin {\u005Cn    opacity: 1 !important\u003B\u005Cn}\u005Cn\u005Cn/* Hide default subtitle */\u005Cn.pf\u002Dc\u002Dlogin__main\u002Dheader\u002Ddesc {\u005Cn    display: none !important\u003B\u005Cn}\u0022, \u0022ui_footer_links\u0022: [], \u0022ui_theme\u0022: \u0022automatic\u0022, \u0022flow_authentication\u0022: \u0022default\u002Dauthentication\u002Dflow\u0022, \u0022flow_invalidation\u0022: \u0022default\u002Dinvalidation\u002Dflow\u0022, \u0022flow_user_settings\u0022: \u0022default\u002Duser\u002Dsettings\u002Dflow\u0022, \u0022default_locale\u0022: \u0022\u0022, \u0022flags\u0022: {\u0022policies_buffered_access_view\u0022: false, \u0022flows_refresh_others\u0022: false}}'),
        versionFamily: "2025.10",
        versionSubdomain: "version-2025-10",
        build: "",
        api: {
            base: "https://auth.single10.app/",
            relBase: "/",
        },
    };
    window.addEventListener("DOMContentLoaded", function () {
        
    });
</script>

<script>
window.authentik.flow = {
    "layout": "stacked",
};
</script>


        
  <meta name="color-scheme" content="light dark" />
  <meta name="theme-color" content="#ffffff" media="(prefers-color-scheme: light)">
  <meta name="theme-color" content="#18191a" media="(prefers-color-scheme: dark)">



        <link rel="stylesheet" type="text/css" href="/static/dist/authentik.css">

        <style>/* Move SSO section above form */
.pf-c-login__main-footer {
    order: -1 !important;
    padding-bottom: 16px !important;
    margin-bottom: 16px !important;
    border-bottom: 1px solid rgba(255,255,255,0.1) !important;
}
.pf-c-login__main-body {
    display: flex !important;
    flex-direction: column !important;
}

/* Make SSO button prominent */
.pf-c-login__main-footer-links-item a {
    display: flex !important;
    align-items: center !important;
    justify-content: center !important;
    gap: 12px !important;
    width: 100% !important;
    padding: 14px 24px !important;
    background: #4285f4 !important;
    color: #fff !important;
    border-radius: 8px !important;
    font-size: 16px !important;
    font-weight: 600 !important;
    text-decoration: none !important;
    transition: background 0.15s !important;
    min-height: 48px !important;
}
.pf-c-login__main-footer-links-item a:hover {
    background: #3367d6 !important;
}
.pf-c-login__main-footer-links-item a img {
    width: 24px !important;
    height: 24px !important;
}
.pf-c-login__main-footer-links {
    display: flex !important;
    flex-direction: column !important;
    padding: 0 !important;
    margin: 0 !important;
    gap: 8px !important;
}
.pf-c-login__main-footer-links-item {
    width: 100% !important;
}

/* Divider */
.pf-c-login__main-footer::after {
    content: "or sign in with email" !important;
    display: block !important;
    text-align: center !important;
    font-size: 13px !important;
    color: rgba(255,255,255,0.4) !important;
    margin-top: 16px !important;
}

/* De-emphasize the password form */
.pf-c-login__main-body .pf-c-form {
    opacity: 0.5 !important;
    transition: opacity 0.2s !important;
}
.pf-c-login__main-body .pf-c-form:hover,
.pf-c-login__main-body .pf-c-form:focus-within {
    opacity: 1 !important;
}

/* Hide default subtitle */
.pf-c-login__main-header-desc {
    display: none !important;
}</style>
        <script src="/static/dist/poly-2025.10.4.js" type="module"></script>
        <script src="/static/dist/standalone/loading/index-2025.10.4.js" type="module"></script>
        
<script src="/static/dist/flow/FlowInterface-2025.10.4.js" type="module"></script>
<style>
:root {
    --ak-flow-background: url("/static/dist/assets/images/flow_background.jpg");
}
</style>

        
        <meta name="sentry-trace" content="f1a20ea9cdf546288ad829cf0d285e1a-a7898a67df2285f4" />
        
        <meta name="baggage" content="" />
        
    </head>
    <body>
        
<ak-skip-to-content></ak-skip-to-content>
<ak-message-container></ak-message-container>
<ak-flow-executor flowSlug="default-authentication-flow">
    <ak-loading></ak-loading>
</ak-flow-executor>

        
        
    </body>
</html>
Sponsored host.tools Pro
Bulk lookups · monitors · webhooks · 10,000 API calls/day

Upgrade to Pro for $19/mo. Cancel anytime. Works with the same API you already use.
How to use robots.txt parser
  1. 1
    Paste your input

    Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.

  2. 2
    Click "Inspect"

    host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.

  3. 3
    Open the API tab

    Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.

Why this matters

Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.

API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fhome.single10.app 
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fhome.single10.app'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
  width="100%" height="600" frameborder="0"></iframe>

Drop into any HTML page. The embed=1 flag hides nav and footer.

FAQ · robots.txt parser

Common questions

Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.
Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.
Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors
  • ✓Schedule any tool — every 1, 5, 15, 60 min
  • ✓Diff against last run, alert on change
  • ✓Webhook + email + Slack + PagerDuty + OpsGenie
  • ✓Bulk CSV upload, 1,000 inputs per job
  • ✓Export results as CSV / NDJSON / Excel
  • ✓90-day history, comparison view