Landmark 2 stops to get here · leads to 6

Activation Function

A non-linear function applied to neuron outputs that introduces non-linearity, enabling networks to learn complex patterns.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.

  3. Activation Function · you are here ✓ understood

Picture it

input xoutputsigmoidtanhReLU
Notice how sigmoid and tanh squash inputs into a range while ReLU passes positives and zeroes negatives: none is a straight line.

Where it sits

Before this

Neural Network
Activation Function

Leads to

Softmax ReLU Sigmoid Vanishing Gradient Maxout Xavier Initialization

Explore nearby

Neural Networks ReLU Rectified Linear Unit - an activation function that outputs the input if positive, zero otherwise. f(x) = max(0, x). Neural Networks Sigmoid An activation function that squashes values to range (0,1), often used for binary classification and gates in LSTMs. Neural Networks Softmax A function that turns a list of scores (logits) into probabilities that are all positive and sum to 1; the standard output of classifiers and language models. Neural Networks GELU Gaussian Error Linear Unit - a smooth activation function combining properties of dropout and ReLU, used in BERT and GPT. Training Vanishing Gradient A problem where gradients become extremely small during backpropagation, preventing deep networks from learning effectively.