#disallow all search agents from crawling the site
User-agent: *
Disallow: /