khunsanuk Posted December 19, 2001 Report Share Posted December 19, 2001 Hi, "Hey, KS, I don't know if you saw it, but IA maintains a page at http://www.archive.org/internet/remove.html to help get pages removed from the archive." That's the page I got all my info from. "I had a look at your robots.txt and it's formatted perfectly." Doesn't really surprise me, I copied it directly of their info page "do you know about the other form of robot exclusion that's done with HTML META tags?" I assume you are talking about stuff like : ? Might not be a bad idea to add this to the board headers. Will look into this. Sanuk! Link to comment Share on other sites More sharing options...
adikgede Posted December 19, 2001 Report Share Posted December 19, 2001 In the event that a spyder doesn't respect robots.txt or NOARCHIVE arguments in your HTML you can always ask your admin to add an entry to your hosts hosts.deny file. Link to comment Share on other sites More sharing options...
Guest Posted December 20, 2001 Report Share Posted December 20, 2001 Post deleted by DoxyBlue Link to comment Share on other sites More sharing options...
khunsanuk Posted December 20, 2001 Report Share Posted December 20, 2001 Hi, At the moment very few (if any) search engines will spider this board. The reason being dynamic pages. The ? should still not pose that many problems, but the & and = do. Sanuk! Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.