# Tells Scanning Robots Where They Are And Are Not Welcome # User-agent: Can also specify by name; "*" is for everyone # Allow: Identifies allowed directories/files # Disallow: If this matches first part of requested path, forget it User-agent: * Disallow: /images/ Disallow: /pdf/ Disallow: /php/ Disallow: /scripts/ User-agent: Googlebot Disallow: /*.pdf$