An attention mask ensuring tokens can only attend to previous positions, crucial for autoregressive generation.
The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.
The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.
Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.
Learning probability distributions over sequences of words to predict what comes next.
A model that generates output one token at a time, using previously generated tokens as input for the next prediction.
Reserved tokens with special meanings like [CLS], [SEP], [MASK], [PAD] used in various model architectures.
A special token used to make sequences the same length in a batch, typically ignored during computation.
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
A collection of data examples used for training, validating, or testing machine learning models.
A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.
A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.
Learning useful features or representations of data automatically, rather than hand-crafting them.
A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.
A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.
A mechanism where each token attends to all other tokens in the sequence to understand contextual relationships.
A binary mask indicating which tokens should be attended to, used to handle padding and causal masking.
An attention mask ensuring tokens can only attend to previous positions, crucial for autoregressive generation.
This concept is essential for understanding large language models and forms a key part of modern AI systems.
Leads to
Nothing yet: a destination in its own right.