• Home
• fi.evolver
• evolver-ai-spring
• 0.18.1
• source code
• Tokenizer.java

×

Project download

Many resources are needed to download a project. Please understand that we have to compensate our server costs. Thank you in advance.
Project price only 1 $

You can buy this project and download/modify it how often you want.

fi.evolver.ai.spring.Tokenizer Maven / Gradle / Ivy

package fi.evolver.ai.spring;

import com.knuddels.jtokkit.api.EncodingType;

public interface Tokenizer {

	public static final Tokenizer CL100K_BASE = JtokkitTokenizer.of(EncodingType.CL100K_BASE);
	
	public static final Tokenizer O200K_BASE = JtokkitTokenizer.of(EncodingType.O200K_BASE);

	public int[] tokenize(String text);

	public default int countTokens(String text) {
		return tokenize(text).length;
	}

}