💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.