Definition
The process of breaking down text into chunks ("tokens") that are used as inputs by LLMs. The phrase "AI is smart" becomes four tokens: "AI", "is", "smart", and punctuation.
Breaking text into chunks for LLM processing
The process of breaking down text into chunks ("tokens") that are used as inputs by LLMs. The phrase "AI is smart" becomes four tokens: "AI", "is", "smart", and punctuation.