Standard 4 stops to get here · leads to 1

Leaky ReLU

A variant of ReLU allowing small negative values (f(x) = x if x > 0, else αx where α ≈ 0.01), preventing dead neurons.

Your route here

4 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.

  3. Activation Function ✓ understood

    A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

  4. ReLU ✓ understood

    Rectified Linear Unit - an activation function that outputs the input if positive, zero otherwise. f(x) = max(0, x).

  5. Leaky ReLU · you are here ✓ understood

A variant of ReLU allowing small negative values (f(x) = x if x > 0, else αx where α ≈ 0.01), preventing dead neurons.

This concept is essential for understanding neural networks & deep learning and forms a key part of modern AI systems.

  • ReLU
  • Activation Function
  • PReLU

Where it sits

Before this

ReLU
Leaky ReLU

Leads to

PReLU

Explore nearby

Neural Networks PReLU Parametric ReLU - like Leaky ReLU but with learnable negative slope parameter. Neural Networks ELU Exponential Linear Unit - an activation function that allows negative values, helping with vanishing gradients. Neural Networks GELU Gaussian Error Linear Unit - a smooth activation function combining properties of dropout and ReLU, used in BERT and GPT. Neural Networks Activation Function A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.