• Home
• org.allenai.nlpstack
• nlpstack-tokenize_2.11
• 1.19
• source code
• WhitespaceTokenizer.scala

×

Project download

Many resources are needed to download a project. Please understand that we have to compensate our server costs. Thank you in advance.
Project price only 1 $

You can buy this project and download/modify it how often you want.

org.allenai.nlpstack.tokenize.WhitespaceTokenizer.scala Maven / Gradle / Ivy

package org.allenai.nlpstack.tokenize

import org.allenai.nlpstack.core.Tokenizer

/* The PennTokenizer was used to tokenize the Penn Treebank.
 * The following is a translation from a sed file.  This algorithm
 * is entirely deterministic.  It is composed of regular expression
 * replacements.
 *
 * @author  Michael Schmitz
 */
object WhitespaceTokenizer extends Tokenizer {
  override def tokenize(string: String) =
    Tokenizer.computeOffsets(string.split("\\s+").toSeq, string)
}