Plucene::Analysis::CharTokenizer.3pm

Langue: en

Autres versions - même langue

Version: 2005-09-05 (ubuntu - 08/07/09)

Section: 3 (Bibliothèques de fonctions)

Sommaire

NAME

Plucene::Analysis::CharTokenizer - base class for character tokenisers

SYNOPSIS

         # isa Plucene::Analysis::Tokenizer
 
         my $next = $chartokenizer->next;

DESCRIPTION

This is an abstract base class for simple, character-oriented tokenizers.

METHODS


token_re

This should be defined in subclasses.

normalize

This will normalise the character before it is added to the token.

next

         my $next = $chartokenizer->next;

This will return the next token in the string, or undef at the end of the string.