#/robots.txt v081001 #/robots.txt file for datanet websites visit: http://www.data.net.uk #/robots.txt help file: http://www.robotstxt.org/wc/faq.html User-agent: * Disallow: /admin Disallow: /secure Disallow: /mail Disallow: /members Disallow: /reports Disallow: /development Disallow: /cgi-bin Disallow: /scripts Disallow: /visuals # this is for legacy sites only Disallow: /visuals.htm # this is for legacy sites only #/stop email gathering User-agent: EmailSiphon Disallow: / User-agent: GRABEMAIL Disallow: / User-agent: GRABEMAIL2 Disallow: /