org.apache.droids.norobots.package.html Maven / Gradle / Ivy

Go to download

Show more of this group Show more artifacts with this name
Show all versions of droids-norobots Show documentation

Apache Droids robots.txt parser

The newest version!





  Using norobots-rfc
   
   Import the class
   
       import org.apache.http.norobots.NoRobotClient;
   
   
   Create an instance for your user-agent
   
       NoRobotClient nrc = NoRobotClient("googlebot");
   
   
   Parse a robots.txt at a site
   
       nrc.parse( new URL( "http://www.apache.org/" ) );
   
   
   Ask if a url is allowed
   
       boolean test = nrc.isUrlAllowed( new URL( "http://www.apache.org/index.html" ) );