Definition
The smallest meaningful unit of text processed by an AI language model, such as a word, sub-word, or punctuation mark.
Smallest meaningful text unit for AI processing
The smallest meaningful unit of text processed by an AI language model, such as a word, sub-word, or punctuation mark.