robots.txt parser
HTTP /api/v1/http/robotsFetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.
https://proxyconfig.corp.google.com/robots.txt
200
8260 bytes
0 User-agent groups
Raw robots.txt
<!--googleoff: all-->
<html lang="en"><head><title>proxyconfig.corp.google.com - Google Single Sign On: Sign into corp</title>
<link href="/c/login.css" rel="stylesheet">
<link rel="icon" href="/c/favicon.ico" type="image/x-icon">
<script type="text/javascript" src="/c/corploginscript.js" nonce="Ec2fIQvocjB6agXsD8BS8Q">
</script>
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
otpParam = "otp";
useOtp = 1;
</script>
<style type="text/css">
h1 {
display: inherit;
font-size: inherit;
margin-block-start: inherit;
margin-block-end: inherit;
margin-inline-start: inherit;
margin-inline-end: inherit;
font-weight: inherit;
}
</style></head>
<body bgcolor="#ffffff" vlink="#666666"><table width="95%" border="0" align="center" cellpadding="0" cellspacing="0" role="presentation"><tr valign="top"><td width="1%"><img src="/c/sso-logo.gif" border="0" align="left" vspace="13" style="height: 3em;" alt="Google Single Sign On"></td>
<td width="99%" bgcolor="#ffffff" valign="top"><table width="100%" cellpadding="1" role="presentation"><tr valign="bottom"><td><div align="right"> </div></td></tr>
<tr><td nowrap><table width="100%" align="center" cellpadding="0" cellspacing="0" bgcolor="#C3D9FF" style="margin-bottom:5" role="presentation"><tr><td class="bubble tl"><img src="/c/tl.gif" alt=""></td>
<th class="bubble" rowspan="2">Single Sign On</th>
<td class="bubble tr"><img src="/c/tr.gif" alt=""></td></tr>
<tr><td class="bubble bl"><img src="/c/bl.gif" alt=""></td>
<td class="bubble br"><img src="/c/br.gif" alt=""></td></tr></table></td></tr></table></td></tr></table>
<br>
<div role="main"><form method="post" id="loginForm" name="loginForm" action="/login"><input type="hidden" id="c" name="c" value="1"/>
<input type="hidden" id="s" name="s" value="proxyconfig.corp.google.com:443/uberproxy/"/>
<input type="hidden" id="d" name="d" value="https://proxyconfig.corp.google.com/robots.txt?upxsrf=AM2vRLn8lyROkGEnuBSrs_bV7oidAGy2ApH5l09chq563gdbrQ%3A1778335750874"/>
<input type="hidden" id="authLevel" name="authLevel" value="2000000"/>
<input type="hidden" id="maxAge" name="maxAge" value="1200"/>
<input type="hidden" id="ssoformat" name="ssoformat" value="CORP_SSO"/>
<input type="hidden" id="interactive" name="interactive" value="yes">
<input type="hidden" id="hasJavascript" name="hasJavascript" value="no">
<input type="hidden" id="mi" name="mi" value="">
<input type="hidden" id="bsc" name="bsc" value="">
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
fillMiField();
</script>
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
const bgChallengeStringFromServer = "";
if (bgChallengeStringFromServer) {
if (window.corploginSetupBotguardWithChallenge) {
window.corploginSetupBotguardWithChallenge(bgChallengeStringFromServer);
}
}
</script>
<table width="1" align="center" cellpadding="5" cellspacing="1" class="signin" role="presentation"><tr><td width="25%" valign="top" style="padding-top: .5em; padding-right: 2em;">
<table cellpadding="5" width="100%" border="0" role="presentation"><tr><td colspan="2" style="text-align:center" nowrap><div class="caption"><h1>Use your SSO username and password</h1></div>
<div class="caption">(* fields are required)</div></td></tr>
<tr><td nowrap><div align="right"><font size="-1" face="Arial, sans-serif"><label for="username">Username*:</label></font></div></td>
<td nowrap><input type="text" name="u" tabindex="1" size="15" id="username" required value="">
<span style="font-size: 83%; position: relative; bottom: 2px;"><span style="padding: 0 3px 0 5px;">@</span>
google.com
<a href="#" id="roleToggle" style="color:black; text-decoration: none" role="button" aria-expanded aria-label="Expand to show role account input">[+]</a>
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
document.getElementById('roleToggle').onclick = function(e) {
e.preventDefault();
toggleInput('roleToggle', 'roleRow', 'role', 2, 1);
}
</script></span></td></tr>
<tr id="roleRow" style="display: none;"><td nowrap><div align="right"><font size="-1" face="Arial, sans-serif"><label for="role">Role:</label></font></div></td>
<td nowrap><input type="text" name="role" size="15" id="role">
<font size="-1"> <a href="https://g3doc.corp.google.com/company/teams/sso/intro.md#role-accounts" tabindex="2">What's this?</a></font></td></tr>
<tr><td nowrap><div align="right"><font size="-1"><label for="password">Password*:</label></font></div></td>
<td nowrap><input type="password" name="pw" tabindex="3" size="15" autocomplete="off" id="password" required></td></tr>
<tr><td><div align="right" style="width: 90px;"><font size="-1"><label for="otp"><a href="/c/SecurityCode.html" tabindex="4" target="_blank" rel="noopener noreferrer">Security Code</a>*:</label></font></div></td>
<td nowrap><input type="text" name="otp" tabindex="4" size="15" autocomplete="off" id="otp" required>
</td></tr>
<tr><td colspan="2" align="center"><input type="submit" name="signInButton" id="signInButton" value="Sign in" style="padding: 2px;" tabindex="4"></td></tr>
<td colspan="2" align="center"></td></table>
<table border="0" width="100%" style="padding-top: 10px;" role="presentation"><tr><td align="left"></td></tr>
<tr><td colspan="2" align="right"><font size="-1"><a href="https://static.corp.google.com/corpsso/techstop/techstop-phones.html">Account Recovery help</a></font></td></tr></table></td>
<td class="image-cell" valign="top"><div id="corplogin-image"><img class="login-image" src="https://static.corp.google.com/corpsso/images/IMG_20200622_173554.jpg" alt=""></div></td></tr></table></form></div>
<div id="interstitial-warning-fedramp" class="fedramp-warning ssobox gnubby-signin hidden"><p>You've successfully signed in and will be redirected in 10 seconds</p>
<p class="errormsg">You didn't use a Titan Security Key as required</p>
<p>Visit <a href="http://go/fedramp-compliance">go/fedramp-compliance</a> to learn why you need to use a Titan Key</p></div>
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
interstitialWarningDurationMs = 10000;
fedrampWarningCookieName = "FEDRAMPWARN";
interstitialWarningCookieName = "INTERSTITIALWARNING";
</script>
<table width="95%" border="0" align="center" cellpadding="0" cellspacing="0" class="footer" role="presentation"><tr valign="top"><td width="99%" bgcolor="#ffffff" valign="top"><table width="100%" cellpadding="1" role="presentation"><tr valign="bottom"><td><div align="right"> </div></td></tr>
<tr><td nowrap><table width="100%" align="center" cellpadding="0" cellspacing="0" bgcolor="#C3D9FF" style="margin-bottom:5" role="presentation"><tr><td class="bubble tl"><img src="/c/tl.gif" alt=""></td>
<th class="bubble" rowspan="2"><span> </span></th>
<td class="bubble tr"><img src="/c/tr.gif" alt=""></td></tr>
<tr><td class="bubble bl"><img src="/c/bl.gif" alt=""></td>
<td class="bubble br"><img src="/c/br.gif" alt=""></td></tr></table></td></tr></table></td>
<td width="1%"><a id="gnubbyLinkFooter" href="https://login.corp.google.com/request?authLevel=2000000&c=1&d=https://proxyconfig.corp.google.com/robots.txt?upxsrf%3DAM2vRLn8lyROkGEnuBSrs_bV7oidAGy2ApH5l09chq563gdbrQ%253A1778335750874&maxAge=1200&s=proxyconfig.corp.google.com:443/uberproxy/"><img src="/c/balls.gif" border="0" alt="Sign in using Security Key" align="left" vspace="13"></a>
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
document.getElementById('gnubbyLinkFooter').onclick = setGnubbyCookie;
</script></td></tr></table>
<script type="text/javascript" nonce="Ec2fIQvocjB6agXsD8BS8Q">
document.getElementById('loginForm').onsubmit = signInOnSubmit;
document.body.onload = function() {
loginOnload();
}
</script></body></html>How to use robots.txt parser
-
1
Paste your input
Enter the value at the top — domain, IP, URL, email, ASN, hash, whatever fits this tool. The smart input auto-detects type.
-
2
Click "Inspect"
host.tools issues real probes (DNS, HTTP, TCP, TLS, WHOIS where applicable) and renders the result in milliseconds.
-
3
Open the API tab
Every web tool has a sibling /api/v1/http/robots JSON endpoint with the same payload. One copy-as-curl click and you're scripting it.
Why this matters
Headers are how the modern web declares its security posture. Auditing them is the highest-ROI thing you can do this week.
API equivalent
/api/v1/http/robots?q=https%3A%2F%2Fproxyconfig.corp.google.com
curl -s '/api/v1/http/robots?q=https%3A%2F%2Fproxyconfig.corp.google.com'
Embed this tool
<iframe src="/http/robots?q={INPUT}&embed=1"
width="100%" height="600" frameborder="0"></iframe>
Drop into any HTML page. The embed=1 flag hides nav and footer.
FAQ · robots.txt parser
Common questions
Is robots.txt parser free?
Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.
Where does the data come from?
Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).
How fresh are the results?
Live by default. Cached for 5 minutes to make repeat queries instant; pass
?nocache=1 for a forced refresh.Can I run this from the command line?
Yes — every tool ships with a copy-as-curl. There's also an official CLI:
host.tools http robots YOUR_INPUT.Can I monitor results over time?
Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.
host.tools Pro
Run robots.txt parser on a schedule. Get pinged when it changes.
Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.
- ✓Schedule any tool — every 1, 5, 15, 60 min
- ✓Diff against last run, alert on change
- ✓Webhook + email + Slack + PagerDuty + OpsGenie
- ✓Bulk CSV upload, 1,000 inputs per job
- ✓Export results as CSV / NDJSON / Excel
- ✓90-day history, comparison view