org.archive.modules.canonicalize.StripWWWRule_en.utf8 Maven / Gradle / Ivy
Go to download
Show more of this group Show more artifacts with this name
Show all versions of heritrix-modules Show documentation
Show all versions of heritrix-modules Show documentation
This project contains some of the configurable modules used within the
Heritrix application to crawl the web. The modules in this project can
be used in applications other than Heritrix, however.
description:
Strip any 'www' found.
Use this rule to equate 'http://www.archive.org/index.html' and
'http://archive.org/index.html'. The resulting canonicalization
returns 'http://archive.org/index.html'. It removes any www's
found, except on URIs that have no path/query component
('slash' pages). Operates on http and https schemes only.
Use the more general StripWWWNRule if you want to strip both 'www'
and 'www01', 'www02', etc.