Short definition
The process of breaking down text into chunks ("tokens") that are used as inputs by LLMs.
Definition
Definition
The process of breaking down text into chunks ("tokens") that are used as inputs by LLMs. The phrase "AI is smart" becomes four tokens: "AI", "is", "smart", and punctuation.