User-agent: * Disallow: /cgi-bin/ Disallow: /dnLoad/ # # Sorry, wget in its recursive mode is a frequent problem. User-agent: wget Disallow: /