Here are some useful resources on what should be in your robots.txt file – which is one of those things I can never seem to find the time for…
- The SKDesigns site has a good (if overly thorough) example robot.txt file
- SearchStatus is a useful FireFox plug-in with a host of SEO related features (inluding the ability to open any site’s robots.txt file in a new tab).
- The basics of robots.txt exclusion can be found robotstxt.org