Standard 10 stops to get here · leads to 1

Masked Language Modeling

A pre-training objective where random tokens are masked and the model learns to predict them from context.

Your route here

10 stops · basics first
  1. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  2. Token ✓ understood

    The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.

  3. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  4. Language Modeling ✓ understood

    Learning probability distributions over sequences of words to predict what comes next.

  5. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  6. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  7. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  8. Unsupervised Learning ✓ understood

    Learning from unlabeled data to discover hidden patterns, structures, or relationships without explicit target outputs.

  9. Self-Supervised Learning ✓ understood

    Learning representations from unlabeled data by creating supervised tasks from the data itself (masked prediction, contrastive learning).

  10. Pre-training ✓ understood

    Training a model on a large dataset (often self-supervised) before fine-tuning on specific tasks, enabling transfer learning.

  11. Masked Language Modeling · you are here ✓ understood

Where it sits

Before this

Language Modeling Pre-training
Masked Language Modeling

Leads to

BERT

Explore nearby

Language & LLMs BERT Bidirectional Encoder Representations from Transformers - a model that understands context by looking at text from both directions. Language & LLMs Causal Language Modeling Training a model to predict the next token given previous tokens, the foundation of autoregressive models like GPT. Language & LLMs Cloze Task A task where words are removed from text and must be predicted, used for evaluation and pre-training. Training Self-Supervised Learning Learning representations from unlabeled data by creating supervised tasks from the data itself (masked prediction, contrastive learning).

In the research

All papers →

A paper that builds on Masked Language Modeling .