No matches.
Free SEO tool
Parse a sitemap and extract all discovered URLs.
Share
Check if a website has a robots.txt file and review its sitemap and disallow rules.
Find all internal links on a webpage.
Checks if a webpage is set to noindex, preventing it from appearing in search engines.
Popular tools
Find plaintext email addresses exposed in a webpage source.
See which AI crawlers the robots.txt of a website allows or blocks for a URL, and which rule decides.
Check the Ahrefs Domain Rating of a website.
Test a URL against any user-agent on the live or a pasted robots.txt, and see the group and line that decide.