robots.txt class

Give this class the contents of a robots.txt file or the URL to it and it will parse it for you. As an option you can give it the name or your robot/spider/crawler and it will return only the section that applies to your case. Works through proxies with the help of Snoopy (see script for more details).

Reviews

There are no comments available yet.

Here you can write a comment


Please enter at least 10 characters.
Loading... Please wait.
* Pflichtangabe
ANG GES G P
A class for maintaining CSV files
ANG GES G P
Yet another directory list class
ANG GES G P
A bunch of classes for a HTTP connection to a webserver
ANG GES G P
template engine for php, allows to store your html code apart from your php code.
ANG GES G P
Sessionara is a handy php class to manage sessions and realize member areas. In a developer's life, the job to make admin areas is a common task ...
ANG GES G P
phPlate is a very simple template program designed to make maintaining a web site as easy as possible. There are zillions of other similar program ...
ANG GES G P
Class for composing and generating XML documents. It features: - Definition of tags and data sections. - Support input and output iso-8859-1 and ...
ANG GES G P
A framework for mysql database searches and displaying the results making it quick and easy to set up a html search form.
949 x viewed
0 x memorized
Details to the advertisement
ANG GES G P
Make a request

You are no longer logged in as a user. Please enter your name and your E-mail-adress!

Loading... Please wait.
Details to the seller

toon.goedhart@