Thanks to a suggestion Paul Boag made on his boagworld podcast, I’ve been listening to and enjoying very much the podcast of the d.construct 2006 conference. The presenters cover some great real-world examples of how people are using API’s like those from Amazon.com, Flickr, and del.ico.us.
Useful resources on what should be in your robots.txt file.
Here are some useful resources on what should be in your robots.txt file – which is one of those things I can never seem to find the time for…
- The SKDesigns site has a good (if overly thorough) example robot.txt file
- SearchStatus is a useful FireFox plug-in with a host of SEO related features (inluding the ability to open any site’s robots.txt file in a new tab).
- The basics of robots.txt exclusion can be found robotstxt.org