Generative AI foundations · Question 51 of 60
A team compares the token count of test reports of equal length in German and English. Which statements are correct?
2 answers are correct. Hard Created with AI
Show answer
Answer
- B The token count determines how much text fits into the context window
- C Texts of equal length can produce different numbers of tokens
Explanation
How a text is split depends on language, word formation and the tokenizer; long compounds often break into several tokens. Since the context window is measured in tokens, counting before large inputs pays off.