User-agent: * Disallow: /cdn-cgi Allow: / Crawl-delay: 1