# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-agent: * Disallow: /vote/ Disallow: /dload/ Disallow: /download/ Disallow: /dl/ Disallow: /downloader/ Sitemap: https://masstamilan.dev/sitemap.xml.gz User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: /