org.archive.modules.canonicalize.StripWWWNRule_en.utf8 Maven / Gradle / Ivy

Go to download

Show more of this group Show more artifacts with this name
Show all versions of heritrix-modules Show documentation

This project contains some of the configurable modules used within the Heritrix application to crawl the web. The modules in this project can be used in applications other than Heritrix, however.

There is a newer version: 3.5.0

Show newest version

description:
Strip any 'www[0-9]*' found.  Use this rule to equate 
'http://www.archive.org/index.html' and 'http://www0001.archive.org/index.html' 
with 'http://archive.org/index.html'.  The resulting canonicalization
returns 'http://archive.org/index.html'.  It removes any www's or wwwNNN's 
found, where 'N' is one or more numerics, EXCEPT on URIs that have no 
path/query component.  Top-level 'slash page' URIs are left unstripped: we 
prefer crawling redundant top pages to missing an entire site only available 
from either the www-full or www-less hostname, but not both.  Operates on http 
and https schemes only.  Use StripWWWRule to strip a lone 'www' only (This rule 
is a more general version of StripWWWRule).