Quote:
Originally Posted by brandrea
Whoa … good work  |
Naaaa, very easy stuff like MS DOS or an Atari BASIC Cartridge :)
robots.txt has been around on the web since, my gosh, feels like forever. My god, I looked it up:
"Part of the Robot Exclusion Protocol, robots. txt was created in 1994 by Martijn Koster after crawlers overwhelmed his website. Though not technically an 'internet standard,' many web developers consider it so, and search engines adopted it to help manage their server resources."
Note that not all search engines / etc follow the webmaster's / server/bandwidth provider's wishes in their robots.txt file.
Quote:
Originally Posted by Paul C
Does this mean Google can't reference TRF?
|
Look at what the robots.txt file says. It is written in plain text, so should be easy to understand :)