Adjusting generation scores based on output length to avoid bias toward shorter or longer sequences.
The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.
The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.
Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.
Learning probability distributions over sequences of words to predict what comes next.
A model that generates output one token at a time, using previously generated tokens as input for the next prediction.
Always selecting the most likely next token during generation, fast but can lead to repetitive or suboptimal outputs.
A generation algorithm that maintains top-k candidates at each step, balancing quality and diversity.
Adjusting generation scores based on output length to avoid bias toward shorter or longer sequences.
This concept is essential for understanding large language models and forms a key part of modern AI systems.