robots.txt class

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Reviews

There are no comments available yet.

Here you can write a comment


Please enter at least 10 characters.
Loading... Please wait.
* Pflichtangabe
ANG GES G P
This class was created to simplify the process of representing images with thumbnails. This class simply gets the image's size, converts the heig ...
ANG GES G P
Form generation and validation class. Uses flat file to describe form including all display options and validation. Validation can be either se ...
ANG GES G P
This is a class to simplify the listing of classes and allows for commandline-like wildcards (i.e. '*.txt'). The class currently returns arrays ...
ANG GES G P
Sessionara is a handy php class to manage sessions and realize member areas. In a developer's life, the job to make admin areas is a common task ...
ANG GES G P
Yet another directory list class
ANG GES G P
This project includes some classes which are used to generate very simple MS Excel file (xls) via PHP. The generated xls file can be obtained by ...
ANG GES G P
Cached Fast Template is a template system for PHP that implements caching for PHP's FastTemplate module. Features include: Supports lock files, so ...
ANG GES G P
I created this class to present the public an easy to use system for protecting pages and register users. Of course there are a lot of more flexi ...
921 x viewed
0 x memorized
Details to the advertisement
ANG GES G P
Make a request

You are no longer logged in as a user. Please enter your name and your E-mail-adress!

Loading... Please wait.
Details to the seller

toon.goedhart@