Self-Attention
A mechanism where each token attends to all other tokens in the sequence to understand contextual relationships.
Related Concepts
- Attention Mechanism: Explore how Attention Mechanism relates to Self-Attention
- Transformer: Explore how Transformer relates to Self-Attention
- Query-Key-Value: Explore how Query-Key-Value relates to Self-Attention
Why It Matters
Understanding Self-Attention is crucial for anyone working with large language models. This concept helps build a foundation for more advanced topics in AI and machine learning.
Learn More
This term is part of the comprehensive AI/ML glossary. Explore related terms to deepen your understanding of this interconnected field.
Tags
Related Terms
Attention Mechanism
A technique that allows neural networks to focus on relevant parts of the input when producing each output, assigning different weights to different input elements.
Query-Key-Value
The three learned projections in attention mechanisms used to compute attention weights and outputs.
Transformer
A neural network architecture introduced in 'Attention is All You Need' (2017) that relies entirely on self-attention mechanisms, becoming the foundation for modern LLMs.