Utils::Tokenizer Class Reference

A utility class for tokenizing strings. More...

#include <Token.hxx>

Public Member Functions
	Tokenizer (string str, string delimiters)
string const &	get_token () const
bool	next_token ()

Protected Attributes
size_t	m_offset
	Current position in the string.
string const	m_string
	The original string to tokenize.
string	m_token
	The current token extracted.
string const	m_delimiters
	Characters used to separate tokens.

Detailed Description

A utility class for tokenizing strings.

!

The Tokenizer class provides a simple and efficient mechanism for breaking a string into smaller components (tokens) based on specified delimiter(s). It facilitates sequential token extraction from the input string and supports custom delimiters.

Usage Example:

Utils::Tokenizer tokenizer("Hello, world! Welcome to Tokenizer.", " ,!");
while (tokenizer.next_token()) {
  std::cout << tokenizer.get_token() << std::endl;
}

Constructor & Destructor Documentation

Tokenizer()

Utils::Tokenizer::Tokenizer ( string str, string delimiters )

inline

Constructs a Tokenizer object with the provided string and delimiters.

Parameters

str	The input string to tokenize.
delimiters	A string containing delimiter characters.

Member Function Documentation

get_token()

string const & Utils::Tokenizer::get_token ( ) const

inline

Retrieves the last extracted token.

Returns

The last token that was retrieved using next_token().

next_token()

bool Utils::Tokenizer::next_token

(

)

Advances the tokenizer to the next token in the string.

This function searches the string for the next token, updating the internal state of the Tokenizer object. If a token is found, it sets the m_token member variable to the new token and returns true. If no more tokens are available, it returns false.

Returns

true if a new token was found; false if no more tokens are available.

Member Data Documentation

m_delimiters

string const Utils::Tokenizer::m_delimiters

protected

Characters used to separate tokens.

m_offset

size_t Utils::Tokenizer::m_offset

protected

Current position in the string.

m_string

string const Utils::Tokenizer::m_string

protected

The original string to tokenize.

m_token

string Utils::Tokenizer::m_token

protected

The current token extracted.

---

The documentation for this class was generated from the following file:

• /Users/enricobertolazzi/Ricerca/PINS/PINS-submodules/UtilsLite/src/Utils/Token.hxx