How generative AI models work · Question 2 of 12
What is a token in a language model?
1 answer is correct. Easy Created with AI
Show answer
Answer
- C A piece of text such as a word, part of a word or punctuation
Explanation
Language models split text into tokens. Common words are often one token, and rare words are split into several parts. In English, a token is about four characters on average. Many models are billed and limited by tokens.