robots.txt parser

HTTP /api/v1/http/robots

Fetch and parse robots.txt — User-agent groups, Disallow/Allow, Sitemap, Crawl-delay.

Try: https://host.com https://checkhost.com https://hostinfo.com https://crypt.tools

Related: HTTP headers Security headers grader Redirect chain tracer SSL certificate inspector robots.txt parser sitemap.xml inspector

https://www.github.com/robots.txt 200 14540 bytes 13 User-agent groups

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

User-agent: *

Allow

/*?tab=achievements&achievement=*
/marketplace/*?tab=readme$
/marketplace/*?tab=transparency$

Disallow

/account-login
/copilot/
/copilot/c/
/ekansa/Open-Context-Data
/ekansa/opencontext-*
/Explodingstuff/
/gist/
/search/advanced
/search$
/*/*/pulse_committer_data.json$
/*/*/pulse_committer_data.json?*
/*/archive/
/*/blame/
/*/cache/
/*/raw/
/*/tree/
*/tarball/
*/zipball/
/.git/
*/.git/
/*.git$
/*.atom$
/*report-abuse?report=*
/*/*/commits/
/marketplace/*/*?tab=readme$
/marketplace/*/*?tab=transparency$
/marketplace/*?*?tab=readme$
/marketplace/*?*?tab=transparency$
/marketplace/actions?tab=readme$
/marketplace/actions?tab=transparency$
/marketplace/free-trials?tab=readme$
/marketplace/free-trials?tab=transparency$
/marketplace/manage?tab=readme$
/marketplace/manage?tab=transparency$
/marketplace/models?tab=readme$
/marketplace/models?tab=transparency$
/marketplace/new?tab=readme$
/marketplace/new?tab=transparency$
/marketplace/publishers?tab=readme$
/marketplace/publishers?tab=transparency$
/marketplace/recently-added?tab=readme$
/marketplace/recently-added?tab=transparency$
/biztools$
/biztools?*
/biztools/*
/businesses$
/businesses?*
/businesses/*
/enterprises$
/enterprises?*
/enterprises/*
/organizations$
/organizations?*
/organizations/*
/settings$
/settings?*
/settings/*
/stafftools$
/stafftools?*
/stafftools/*
/users/*/billing$
/users/*/billing?*
/users/*/billing/*
/writing$
/writing?*
/writing/*
/*/*/branches$
/*/*/branches?*
/*/*/branches/*
/*/*/comments$
/*/*/comments?*
/*/*/comments/*
/*/*/compare$
/*/*/compare?*
/*/*/compare/*
/*/*/contributors$
/*/*/contributors?*
/*/*/contributors/*
/*/*/contributors_list$
/*/*/contributors_list?*
/*/*/contributors_list/*
/*/*/contributors-list$
/*/*/contributors-list?*
/*/*/contributors-list/*
/*/*/download$
/*/*/download?*
/*/*/download/*
/*/*/forks$
/*/*/forks?*
/*/*/forks/*
/*/*/graphs$
/*/*/graphs?*
/*/*/graphs/*
/*/*/issues/new$
/*/*/issues/new?*
/*/*/issues/new/*
/*/*/issues/search$
/*/*/issues/search?*
/*/*/issues/search/*
/*/*/milestones/new$
/*/*/milestones/new?*
/*/*/milestones/new/*
/*/*/network$
/*/*/network?*
/*/*/network/*
/*/*/projects$
/*/*/projects?*
/*/*/projects/*
/*/*/pulse$
/*/*/pulse?*
/*/*/pulse/*
/*/*/pulse-new$
/*/*/pulse-new?*
/*/*/pulse-new/*
/*/*/pulse_committer_data$
/*/*/pulse_committer_data?*
/*/*/pulse_committer_data/*
/*/*/pulse_diffstat_summary$
/*/*/pulse_diffstat_summary?*
/*/*/pulse_diffstat_summary/*
/*/*/revisions$
/*/*/revisions?*
/*/*/revisions/*
/*/*/stargazers$
/*/*/stargazers?*
/*/*/stargazers/*
/*/*/tags$
/*/*/tags?*
/*/*/tags/*
/*/*/watchers$
/*/*/watchers?*
/*/*/watchers/*
/*?source=*
/*&source=*
/*?ref_cta=*
/*&ref_cta=*
/*?plan=*
/*&plan=*
/*?return_to=*
/*&return_to=*
/*?ref_loc=*
/*&ref_loc=*
/*?setup_organization=*
/*&setup_organization=*
/*?source_repo=*
/*&source_repo=*
/*?ref_page=*
/*&ref_page=*
/*?referrer=*
/*&referrer=*
/*?report=*
/*&report=*
/*?author=*
/*&author=*
/*?since=*
/*&since=*
/*?until=*
/*&until=*
/*?tab=*
/*&tab=*
/*?q=*
/*&q=*

Raw robots.txt

# If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots
# We also provide an extensive API: https://docs.github.com
Sitemap: https://github.com/sitemap_marketing.xml

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
Crawl-delay: 1
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$
Disallow: /marketplace/new?tab=readme$
Disallow: /marketplace/new?tab=transparency$
Disallow: /marketplace/publishers?tab=readme$
Disallow: /marketplace/publishers?tab=transparency$
Disallow: /marketplace/recently-added?tab=readme$
Disallow: /marketplace/recently-added?tab=transparency$
Disallow: /biztools$
Disallow: /biztools?*
Disallow: /biztools/*
Disallow: /businesses$
Disallow: /businesses?*
Disallow: /businesses/*
Disallow: /enterprises$
Disallow: /enterprises?*
Disallow: /enterprises/*
Disallow: /organizations$
Disallow: /organizations?*
Disallow: /organizations/*
Disallow: /settings$
Disallow: /settings?*
Disallow: /settings/*
Disallow: /stafftools$
Disallow: /stafftools?*
Disallow: /stafftools/*
Disallow: /users/*/billing$
Disallow: /users/*/billing?*
Disallow: /users/*/billing/*
Disallow: /writing$
Disallow: /writing?*
Disallow: /writing/*
Disallow: /*/*/branches$
Disallow: /*/*/branches?*
Disallow: /*/*/branches/*
Disallow: /*/*/comments$
Disallow: /*/*/comments?*
Disallow: /*/*/comments/*
Disallow: /*/*/compare$
Disallow: /*/*/compare?*
Disallow: /*/*/compare/*
Disallow: /*/*/contributors$
Disallow: /*/*/contributors?*
Disallow: /*/*/contributors/*
Disallow: /*/*/contributors_list$
Disallow: /*/*/contributors_list?*
Disallow: /*/*/contributors_list/*
Disallow: /*/*/contributors-list$
Disallow: /*/*/contributors-list?*
Disallow: /*/*/contributors-list/*
Disallow: /*/*/download$
Disallow: /*/*/download?*
Disallow: /*/*/download/*
Disallow: /*/*/forks$
Disallow: /*/*/forks?*
Disallow: /*/*/forks/*
Disallow: /*/*/graphs$
Disallow: /*/*/graphs?*
Disallow: /*/*/graphs/*
Disallow: /*/*/issues/new$
Disallow: /*/*/issues/new?*
Disallow: /*/*/issues/new/*
Disallow: /*/*/issues/search$
Disallow: /*/*/issues/search?*
Disallow: /*/*/issues/search/*
Disallow: /*/*/milestones/new$
Disallow: /*/*/milestones/new?*
Disallow: /*/*/milestones/new/*
Disallow: /*/*/network$
Disallow: /*/*/network?*
Disallow: /*/*/network/*
Disallow: /*/*/projects$
Disallow: /*/*/projects?*
Disallow: /*/*/projects/*
Disallow: /*/*/pulse$
Disallow: /*/*/pulse?*
Disallow: /*/*/pulse/*
Disallow: /*/*/pulse-new$
Disallow: /*/*/pulse-new?*
Disallow: /*/*/pulse-new/*
Disallow: /*/*/pulse_committer_data$
Disallow: /*/*/pulse_committer_data?*
Disallow: /*/*/pulse_committer_data/*
Disallow: /*/*/pulse_diffstat_summary$
Disallow: /*/*/pulse_diffstat_summary?*
Disallow: /*/*/pulse_diffstat_summary/*
Disallow: /*/*/revisions$
Disallow: /*/*/revisions?*
Disallow: /*/*/revisions/*
Disallow: /*/*/stargazers$
Disallow: /*/*/stargazers?*
Disallow: /*/*/stargazers/*
Disallow: /*/*/tags$
Disallow: /*/*/tags?*
Disallow: /*/*/tags/*
Disallow: /*/*/watchers$
Disallow: /*/*/watchers?*
Disallow: /*/*/watchers/*
Disallow: /*?source=*
Disallow: /*&source=*
Disallow: /*?ref_cta=*
Disallow: /*&ref_cta=*
Disallow: /*?plan=*
Disallow: /*&plan=*
Disallow: /*?return_to=*
Disallow: /*&return_to=*
Disallow: /*?ref_loc=*
Disallow: /*&ref_loc=*
Disallow: /*?setup_organization=*
Disallow: /*&setup_organization=*
Disallow: /*?source_repo=*
Disallow: /*&source_repo=*
Disallow: /*?ref_page=*
Disallow: /*&ref_page=*
Disallow: /*?referrer=*
Disallow: /*&referrer=*
Disallow: /*?report=*
Disallow: /*&report=*
Disallow: /*?author=*
Disallow: /*&author=*
Disallow: /*?since=*
Disallow: /*&since=*
Disallow: /*?until=*
Disallow: /*&until=*
Disallow: /*?tab=*
Disallow: /*&tab=*
Disallow: /*?q=*
Disallow: /*&q=*

User-agent: Bytespider
Disallow: /

User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$
Disallow: /marketplace/new?tab=readme$
Disallow: /marketplace/new?tab=transparency$
Disallow: /marketplace/publishers?tab=readme$
Disallow: /marketplace/publishers?tab=transparency$
Disallow: /marketplace/recently-added?tab=readme$
Disallow: /marketplace/recently-added?tab=transparency$
Disallow: /biztools$
Disallow: /biztools?*
Disallow: /biztools/*
Disallow: /businesses$
Disallow: /businesses?*
Disallow: /businesses/*
Disallow: /enterprises$
Disallow: /enterprises?*
Disallow: /enterprises/*
Disallow: /organizations$
Disallow: /organizations?*
Disallow: /organizations/*
Disallow: /settings$
Disallow: /settings?*
Disallow: /settings/*
Disallow: /stafftools$
Disallow: /stafftools?*
Disallow: /stafftools/*
Disallow: /users/*/billing$
Disallow: /users/*/billing?*
Disallow: /users/*/billing/*
Disallow: /writing$
Disallow: /writing?*
Disallow: /writing/*
Disallow: /*/*/branches$
Disallow: /*/*/branches?*
Disallow: /*/*/branches/*
Disallow: /*/*/comments$
Disallow: /*/*/comments?*
Disallow: /*/*/comments/*
Disallow: /*/*/compare$
Disallow: /*/*/compare?*
Disallow: /*/*/compare/*
Disallow: /*/*/contributors$
Disallow: /*/*/contributors?*
Disallow: /*/*/contributors/*
Disallow: /*/*/contributors_list$
Disallow: /*/*/contributors_list?*
Disallow: /*/*/contributors_list/*
Disallow: /*/*/contributors-list$
Disallow: /*/*/contributors-list?*
Disallow: /*/*/contributors-list/*
Disallow: /*/*/download$
Disallow: /*/*/download?*
Disallow: /*/*/download/*
Disallow: /*/*/forks$
Disallow: /*/*/forks?*
Disallow: /*/*/forks/*
Disallow: /*/*/graphs$
Disallow: /*/*/graphs?*
Disallow: /*/*/graphs/*
Disallow: /*/*/issues/new$
Disallow: /*/*/issues/new?*
Disallow: /*/*/issues/new/*
Disallow: /*/*/issues/search$
Disallow: /*/*/issues/search?*
Disallow: /*/*/issues/search/*
Disallow: /*/*/milestones/new$
Disallow: /*/*/milestones/new?*
Disallow: /*/*/milestones/new/*
Disallow: /*/*/network$
Disallow: /*/*/network?*
Disallow: /*/*/network/*
Disallow: /*/*/projects$
Disallow: /*/*/projects?*
Disallow: /*/*/projects/*
Disallow: /*/*/pulse$
Disallow: /*/*/pulse?*
Disallow: /*/*/pulse/*
Disallow: /*/*/pulse-new$
Disallow: /*/*/pulse-new?*
Disallow: /*/*/pulse-new/*
Disallow: /*/*/pulse_committer_data$
Disallow: /*/*/pulse_committer_data?*
Disallow: /*/*/pulse_committer_data/*
Disallow: /*/*/pulse_diffstat_summary$
Disallow: /*/*/pulse_diffstat_summary?*
Disallow: /*/*/pulse_diffstat_summary/*
Disallow: /*/*/revisions$
Disallow: /*/*/revisions?*
Disallow: /*/*/revisions/*
Disallow: /*/*/stargazers$
Disallow: /*/*/stargazers?*
Disallow: /*/*/stargazers/*
Disallow: /*/*/tags$
Disallow: /*/*/tags?*
Disallow: /*/*/tags/*
Disallow: /*/*/watchers$
Disallow: /*/*/watchers?*
Disallow: /*/*/watchers/*
Disallow: /*?source=*
Disallow: /*&source=*
Disallow: /*?ref_cta=*
Disallow: /*&ref_cta=*
Disallow: /*?plan=*
Disallow: /*&plan=*
Disallow: /*?return_to=*
Disallow: /*&return_to=*
Disallow: /*?ref_loc=*
Disallow: /*&ref_loc=*
Disallow: /*?setup_organization=*
Disallow: /*&setup_organization=*
Disallow: /*?source_repo=*
Disallow: /*&source_repo=*
Disallow: /*?ref_page=*
Disallow: /*&ref_page=*
Disallow: /*?referrer=*
Disallow: /*&referrer=*
Disallow: /*?report=*
Disallow: /*&report=*
Disallow: /*?author=*
Disallow: /*&author=*
Disallow: /*?since=*
Disallow: /*&since=*
Disallow: /*?until=*
Disallow: /*&until=*
Disallow: /*?tab=*
Disallow: /*&tab=*
Disallow: /*?q=*
Disallow: /*&q=*

User-agent: bingbot
User-agent: adidxbot
User-agent: BingPreview
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /account-login
Disallow: */tarball/
Disallow: */zipball/
Disallow: /Explodingstuff/
Disallow: /copilot/
Disallow: /copilot/c/

User-agent: baidu
crawl-delay: 1

User-agent: *
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$
Disallow: /marketplace/new?tab=readme$
Disallow: /marketplace/new?tab=transparency$
Disallow: /marketplace/publishers?tab=readme$
Disallow: /marketplace/publishers?tab=transparency$
Disallow: /marketplace/recently-added?tab=readme$
Disallow: /marketplace/recently-added?tab=transparency$
Disallow: /biztools$
Disallow: /biztools?*
Disallow: /biztools/*
Disallow: /businesses$
Disallow: /businesses?*
Disallow: /businesses/*
Disallow: /enterprises$
Disallow: /enterprises?*
Disallow: /enterprises/*
Disallow: /organizations$
Disallow: /organizations?*
Disallow: /organizations/*
Disallow: /settings$
Disallow: /settings?*
Disallow: /settings/*
Disallow: /stafftools$
Disallow: /stafftools?*
Disallow: /stafftools/*
Disallow: /users/*/billing$
Disallow: /users/*/billing?*
Disallow: /users/*/billing/*
Disallow: /writing$
Disallow: /writing?*
Disallow: /writing/*
Disallow: /*/*/branches$
Disallow: /*/*/branches?*
Disallow: /*/*/branches/*
Disallow: /*/*/comments$
Disallow: /*/*/comments?*
Disallow: /*/*/comments/*
Disallow: /*/*/compare$
Disallow: /*/*/compare?*
Disallow: /*/*/compare/*
Disallow: /*/*/contributors$
Disallow: /*/*/contributors?*
Disallow: /*/*/contributors/*
Disallow: /*/*/contributors_list$
Disallow: /*/*/contributors_list?*
Disallow: /*/*/contributors_list/*
Disallow: /*/*/contributors-list$
Disallow: /*/*/contributors-list?*
Disallow: /*/*/contributors-list/*
Disallow: /*/*/download$
Disallow: /*/*/download?*
Disallow: /*/*/download/*
Disallow: /*/*/forks$
Disallow: /*/*/forks?*
Disallow: /*/*/forks/*
Disallow: /*/*/graphs$
Disallow: /*/*/graphs?*
Disallow: /*/*/graphs/*
Disallow: /*/*/issues/new$
Disallow: /*/*/issues/new?*
Disallow: /*/*/issues/new/*
Disallow: /*/*/issues/search$
Disallow: /*/*/issues/search?*
Disallow: /*/*/issues/search/*
Disallow: /*/*/milestones/new$
Disallow: /*/*/milestones/new?*
Disallow: /*/*/milestones/new/*
Disallow: /*/*/network$
Disallow: /*/*/network?*
Disallow: /*/*/network/*
Disallow: /*/*/projects$
Disallow: /*/*/projects?*
Disallow: /*/*/projects/*
Disallow: /*/*/pulse$
Disallow: /*/*/pulse?*
Disallow: /*/*/pulse/*
Disallow: /*/*/pulse-new$
Disallow: /*/*/pulse-new?*
Disallow: /*/*/pulse-new/*
Disallow: /*/*/pulse_committer_data$
Disallow: /*/*/pulse_committer_data?*
Disallow: /*/*/pulse_committer_data/*
Disallow: /*/*/pulse_diffstat_summary$
Disallow: /*/*/pulse_diffstat_summary?*
Disallow: /*/*/pulse_diffstat_summary/*
Disallow: /*/*/revisions$
Disallow: /*/*/revisions?*
Disallow: /*/*/revisions/*
Disallow: /*/*/stargazers$
Disallow: /*/*/stargazers?*
Disallow: /*/*/stargazers/*
Disallow: /*/*/tags$
Disallow: /*/*/tags?*
Disallow: /*/*/tags/*
Disallow: /*/*/watchers$
Disallow: /*/*/watchers?*
Disallow: /*/*/watchers/*
Disallow: /*?source=*
Disallow: /*&source=*
Disallow: /*?ref_cta=*
Disallow: /*&ref_cta=*
Disallow: /*?plan=*
Disallow: /*&plan=*
Disallow: /*?return_to=*
Disallow: /*&return_to=*
Disallow: /*?ref_loc=*
Disallow: /*&ref_loc=*
Disallow: /*?setup_organization=*
Disallow: /*&setup_organization=*
Disallow: /*?source_repo=*
Disallow: /*&source_repo=*
Disallow: /*?ref_page=*
Disallow: /*&ref_page=*
Disallow: /*?referrer=*
Disallow: /*&referrer=*
Disallow: /*?report=*
Disallow: /*&report=*
Disallow: /*?author=*
Disallow: /*&author=*
Disallow: /*?since=*
Disallow: /*&since=*
Disallow: /*?until=*
Disallow: /*&until=*
Disallow: /*?tab=*
Disallow: /*&tab=*
Disallow: /*?q=*
Disallow: /*&q=*

Common questions

Is robots.txt parser free?

Yes — every tool is free on the web with a 200/hour rate limit per IP. The matching API endpoint /api/v1/http/robots is free up to 100 requests/hour, no key required.

Where does the data come from?

Real-time probes against authoritative sources (DNS root, RIRs, registries, the target server itself), plus partner data feeds from hostinfo.com (GeoIP/ASN) and hostcheck.com (reputation).

How fresh are the results?

Live by default. Cached for 5 minutes to make repeat queries instant; pass ?nocache=1 for a forced refresh.

Can I run this from the command line?

Yes — every tool ships with a copy-as-curl. There's also an official CLI: host.tools http robots YOUR_INPUT.

Can I monitor results over time?

Pro tier lets you schedule any tool to run every 1/5/15/60 min and alert on diff. See monitors.

host.tools Pro

Run robots.txt parser on a schedule. Get pinged when it changes.

Pro gets you bulk lookups, monitors, webhook alerts, history, exports and 10,000 API calls/day. $19/mo.

See pricing Tour monitors

✓Schedule any tool — every 1, 5, 15, 60 min
✓Diff against last run, alert on change
✓Webhook + email + Slack + PagerDuty + OpsGenie
✓Bulk CSV upload, 1,000 inputs per job
✓Export results as CSV / NDJSON / Excel
✓90-day history, comparison view