User-Agent: LinkChecker Allow: / User-agent: * disallow: form_upload disallow: /blank.html disallow: /notfound.html disallow: /privacy.html disallow: /terms-of-use.html User-Agent: MJ12bot Crawl-Delay: 5