Basic GenAI concepts · Question 1 of 12
What is a token for a language model?
1 answer is correct. Easy
Show answer
Answer
- A A word or part of a word as a unit of text
Explanation
Language models split text into tokens, which are words, parts of words or punctuation marks. Input and output are measured in tokens, as are the limit of the context window and usually the price. One word can consist of several tokens.