robots.txt class

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Reviews

There are no comments available yet.

Here you can write a comment


Please enter at least 10 characters.
Loading... Please wait.
* Pflichtangabe
ANG GES G P
Simple classe to display a recordset with a page next and previous. You could customize it like google.com :)
ANG GES G P
Cached Fast Template is a template system for PHP that implements caching for PHP's FastTemplate module. Features include: Supports lock files, so ...
ANG GES G P
The HtmlSource class is a simple way of retrieving, and stripping, HTML source from any website, given a URL. Supports GET and POST and also coo ...
ANG GES G P
A generalized parsed template caching mechanism, that can work with a variety of existing template classes.
ANG GES G P
A PHP library to do full text indexing with MySQL
ANG GES G P
Diese Klasse unterstützt alle MySQL Datenbanken ab der V3.x aufwärts und läuft unter PHP4 und PHP5. Eine leicht erlernbare API sollte es auch A ...
ANG GES G P
This is a little improvement to "Mp3 & ID3 tag parser" by marty sejba . I made this to support ID3v1.1 TAGS (with track number info) and added ...
ANG GES G P
OIS ermöglicht Oracle DBAs den schnellen Zugriff auf diverse Oracle Parameter und Tuning Views, unter anderem werden folgende Informationen angez ...
940 x viewed
0 x memorized
Details to the advertisement
ANG GES G P
Make a request

You are no longer logged in as a user. Please enter your name and your E-mail-adress!

Loading... Please wait.
Details to the seller

toon.goedhart@