robots.txt class

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Reviews

There are no comments available yet.

Here you can write a comment


Please enter at least 10 characters.
Loading... Please wait.
* Pflichtangabe
ANG GES G P
Simple and easy class for reading/writing 3dim arrays as CSV files.
ANG GES G P
fileupload.class extends PHP's native support for uploading image and text files through a Web browser. It can clean up the file's name, detect it ...
ANG GES G P
Session management with timeout and applications, based on temporary files
ANG GES G P
Object manipulation routine.
ANG GES G P
Diese Template-Klasse wurde für kleine Webprojekte unter dem Gesichtspunkt entwickelt, dass die Benutzung so einfach wie nur irgendwie möglich s ...
ANG GES G P
The LDAP class for PHP is an easy way to interface your PHP program with an LDAP database. The interface is similar to what you would find in a fi ...
ANG GES G P
Simple class to translate strings into UTF-8 from your favorite charset.
ANG GES G P
Ever written some large-ish PHP app and wished you didn't have to write all those SQL statements by hand? To create all those oh-so-similar classe ...
732 x viewed
0 x memorized
Details to the advertisement
ANG GES G P
Make a request

You are no longer logged in as a user. Please enter your name and your E-mail-adress!

Loading... Please wait.
Details to the seller

toon.goedhart@