#User-agent: Scrapy #Allow: / User-agent: * Disallow: Sitemap: https://www.fdm4.com/sitemap_index.xml