Generative AI foundations · Question 27 of 60
Which statements about tokens are correct?
2 answers are correct. Medium Created with AI
Show answer
Answer
- B A token can be a single character
- C A token can be part of a word
Explanation
Tokens range from single characters through parts of words to whole words. Punctuation also becomes tokens. That is why the token count cannot simply be read from the word count.