Neural Networks & Deep Learning
Leaky ReLU
A variant of ReLU allowing small negative values (f(x) = x if x > 0, else αx where α ≈ 0.01), preventing dead neurons.
This concept is essential for understanding neural networks & deep learning and forms a key part of modern AI systems.
Related Concepts
- ReLU
- Activation Function
- PReLU
Tags
neural-networks-deep-learning relu activation-function prelu
Related Terms
Activation Function
A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.
PReLU
Parametric ReLU - like Leaky ReLU but with learnable negative slope parameter.
ReLU
Rectified Linear Unit - an activation function that outputs the input if positive, zero otherwise. f(x) = max(0, x).