Standard 3 stops to get here · leads to 6
ReLU
Rectified Linear Unit - an activation function that outputs the input if positive, zero otherwise. f(x) = max(0, x).
Your route here
3 stops · basics first
- Machine Learning ✓ understood
Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.
- Neural Network ✓ understood
A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.
- Activation Function ✓ understood
A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.
- ReLU · you are here ✓ understood
Where it sits
Explore nearby
Neural Networks Leaky ReLU A variant of ReLU allowing small negative values (f(x) = x if x > 0, else αx where α ≈ 0.01), preventing dead neurons. Neural Networks GELU Gaussian Error Linear Unit - a smooth activation function combining properties of dropout and ReLU, used in BERT and GPT. Neural Networks ELU Exponential Linear Unit - an activation function that allows negative values, helping with vanishing gradients. Training Vanishing Gradient A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively. Neural Networks He Initialization Weight initialization designed for ReLU activations, preventing vanishing/exploding gradients in deep networks.