hrmph!
i was forced to take down a section of the web that has been there for at least 5 years that i can recall, and probably a lot longer than that, because the Caliphate O.T.O.’s web crawler finally figured out where i had stashed Liber CDXIV. in spite of all their bluster, they haven’t been able to prove that they actually own the copyright to a piece that should be in the public domain by now anyway, but i’ve had to take it down because i can’t afford to defend myself against a group who thinks it’s okay to violate uncle al’s specific request that people involved in the great work not file lawsuits against one another… feh!
gotcha… thanks.
I think it’s documented in most pages about robots.txt, but you have to know what you’re looking at. For example, in this page, it starts out “Search engine robots will check a special file in the root of each server called robots.txt” – note that it says “root of each server” not each or every directory. There are probably pages out there which are more explicit, as I know some discuss when to use robots.txt and when to use META tags…
is that documented somewhere? i don’t disbelieve you, i just want to get it from the source, so to speak… and i’m definitely going to put meta tags in now… 8/
That’s outside the definition. robots.txt is ONLY loaded from the server root. It has to be at http://www.drizzle.com/robots.txt … that’s the only place spiders will look for it. If you don’t have access to server root, you have to use meta-tags.
the whole ~slmndr directory (of which uncle_al is a child) has a disallow for user-agent:* in robots.txt… that’s what really frustrates me about it…
Nothing is safe from Google. You should use a no-index META tag…
i already have a link to that in my bookmarks, but the problem i have with this is that i didn’t link it anywhere, and i didn’t tell anyone about it, and still, somehow it managed to make it to #1 on google, and #6 on yahoo when you searched for “Liber CDXIV”… my guess is that somebody else found it accidentally and linked to it, and it spread from there, but it’s frustrating… and even more so because they should be fighting with each other about things that have more significance than who owns the copyright to material that should be in the public domain anyway.
Then you’ll want to put up a link to the paraphrased version, The Magick Arte…