Rechercher une page de manuel
HCT::Lang::Token.3pm
Langue: en
Version: 2009-08-08 (fedora - 01/12/10)
Section: 3 (Bibliothèques de fonctions)
NAME
HCT::Lang::Token - Definition of tokens.DESCRIPTION
A token is a categorized block of text. The block of text corresponding to the token is known as a lexeme. A lexical analyzer divides a sequence of characters into tokens (tokenization) and categorizes them according to function, giving them meaning. A token can look like anything; it just needs to be a useful part of the structured text.The object of token created by class "HCT::Lang::Token" has such features: name, type and list of patterns. The lexemes will be gotten by lexer by using patterns.
The token name is an unique name that will be returned by lexer to parser.
The patterns are a list of patterns, that will be used for definition of the token name and token lexeme. All patterns described by regular expressions.
The token type is a number, that means position/preority, where the token have to be used. For instance, tokens can be divided on keywords, operators, numbers and identifiers, and the order in which lexer should use them will be: operators, numbers, keywords and identifiers. According to this the order can be: operators - 3, numbers - 2, keywrods - 1, identifiers - 0.
FUNCTIONS
The functions above will be exported by default.- EMPTY_TOKEN
- When the lexer reach the end of input, it returns the '' empty token with an "undef" value.
CONSTRUCTOR
- new ()
- Creates a new "HCT::Language::Token" object.
METHODS
- get_patterns ()
- Returns a list of token patterns.
- get_name ()
- Returns token name.
- get_type ()
- Returns token type.
Contenus ©2006-2024 Benjamin Poulain
Design ©2006-2024 Maxime Vantorre