Being A Good Internet Citizen

Reading Mark Pilgrim's discussion on the importance of respecting robots.txt has sparked my interest in the importance of being a good internet citizen. First off, why on Earth would any news aggregator purposely download the entire content of an item? I thought the purpose of an aggregator was so that the user could decide whether to read a given item, then take the appropriate steps to do so. Second, the abuse of the refer header (which has been well documented and is being rectified) is something that needs to be addressed in other clients.

Why the (seemingly) sudden interest in such implementation details? I have my reasons...

Watch this space. There's going to be some fun soon.

February 20, 2003 - 08:57 PM