Short definition
The smallest meaningful unit of text processed by an AI language model, such as a word, sub-word, or punctuation mark.
Definition
Definition
The smallest meaningful unit of text processed by an AI language model, such as a word, sub-word, or punctuation mark.