Generative AI foundations · Question 6 of 60
What is a token in a language model?
1 answer is correct. Easy Created with AI
Show answer
Answer
- A A small unit of text, such as part of a word
Explanation
Tokenization splits text into smaller units. A token can be a single character, part of a word or a whole word. An API access key is colloquially also called a token, but that is not what is meant here.