robots.txt class

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Reviews

There are no comments available yet.

Here you can write a comment


Please enter at least 10 characters.
Loading... Please wait.
* Pflichtangabe
ANG GES G P
Ever written some large-ish PHP app and wished you didn't have to write all those SQL statements by hand? To create all those oh-so-similar classe ...
ANG GES G P
simple implementation of the nntp protocol which allows php to make postings to a newsgroup. You can also reply to your own postngs on the same ...
ANG GES G P
Die PHP-interne Funktion parse_url reicht nicht immer aus, um URLs bzw. URIs korrekt in ihre Komponenten zu zerlegen, wie sich u.a. im PHP- ...
ANG GES G P
phpSorm is mainly a base-class which provides a simple way to map a table from a relational database to a PHP-object. The extended functionality i ...
ANG GES G P
Simple classe to display a recordset with a page next and previous. You could customize it like google.com :)
ANG GES G P
entspricht Version 2.1 der vCard-Spezifikation (http://www.imc.org/pdi/) erlaubt folgende Felder: Name Formatted Name Telefon-/Faxnummern ...
ANG GES G P
Extension for the forms generation and validation class to manage forms that are presented in multiple pages. It features: - Sequential mode (Wi ...
ANG GES G P
A class for use with ODP (dmoz.org) data retrieval.
1169 x viewed
0 x memorized
Details to the advertisement
ANG GES G P
Make a request

You are no longer logged in as a user. Please enter your name and your E-mail-adress!

Loading... Please wait.
Details to the seller

toon.goedhart@