How generative AI models work · Question 2 of 12

What is a token in a language model?

1 answer is correct. Easy Created with AI

Show answer

Answer

  • C A piece of text such as a word, part of a word or punctuation

Explanation

Language models split text into tokens. Common words are often one token, and rare words are split into several parts. In English, a token is about four characters on average. Many models are billed and limited by tokens.

Certifications & exams

Which exam comes next?

You can already practise for 51 exams, and 16 are up for a vote. See all certifications with their official information, cast your vote or suggest your exam.

See all certifications