Fundamentals
Tokenization
The process of splitting text into tokens before a model can process it. Different models use different tokenizers, so the same text can cost a slightly different number of tokens on each.
In practice
The word "tokenization" might split into "token" + "ization", counting as 2 tokens instead of 1.
Related terms
See what your tokens really cost
Track usage and spend across every model and platform, free.
Image: Google DeepMind on Pexels. Definition free to reuse under CC BY 4.0.