A technique reducing the likelihood of previously generated tokens to avoid repetitive outputs.
The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.
The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.
Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.
Learning probability distributions over sequences of words to predict what comes next.
Automatically creating coherent text using language models, from simple completion to creative writing.
A technique reducing the likelihood of previously generated tokens to avoid repetitive outputs.
This concept is essential for understanding large language models and forms a key part of modern AI systems.
Before this
Text GenerationLeads to
Nothing yet: a destination in its own right.