Large Language Models
Attention Score
The weight determining how much each value contributes to the output, computed from query-key similarity.
This concept is essential for understanding large language models and forms a key part of modern AI systems.
Related Concepts
- Attention Mechanism
- Query-Key-Value
- Softmax
Tags
large-language-models attention-mechanism query-key-value softmax
Related Terms
Attention Mechanism
A technique that allows neural networks to focus on relevant parts of the input when producing each output, assigning different weights to different input elements.
Query-Key-Value
The three learned projections in attention mechanisms used to compute attention weights and outputs.
Softmax
An activation function that converts a vector of values into a probability distribution, commonly used for multi-class classification.