Landmark 3 stops to get here · leads to 7

Language Modeling

Learning probability distributions over sequences of words to predict what comes next.

Your route here

3 stops · basics first
  1. Natural Language Processing ✓ understood

    The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.

  2. Token ✓ understood

    The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.

  3. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  4. Language Modeling · you are here ✓ understood

Picture it

  1. 01 Context "The cat sat on the"
  2. 02 Language model Reads all previous tokens
  3. 03 Next-token distribution P(mat) 0.41, P(floor) 0.22, P(sofa) 0.09…
  4. 04 Pick a token Greedy or sampled: "mat"
  5. 05 Append and repeat The new token joins the context
Notice the model never outputs a sentence directly: it outputs a probability for every possible next token, one step at a time.

Learning probability distributions over sequences of words to predict what comes next.

This concept is essential for understanding natural language processing and forms a key part of modern AI systems.

  • LLM
  • NLP
  • Next Token Prediction

Where it sits

Before this

Natural Language Processing Tokenization
Language Modeling

Leads to

Large Language Model Autoregressive Model Causal Language Modeling Masked Language Modeling Perplexity Text Generation

+ 1 more

Explore nearby

Language & LLMs Causal Language Modeling Training a model to predict the next token given previous tokens, the foundation of autoregressive models like GPT. Language & LLMs Masked Language Modeling A pre-training objective where random tokens are masked and the model learns to predict them from context. Evaluation Perplexity A metric measuring how well a language model predicts text - lower perplexity indicates better prediction. Language & LLMs N-gram A contiguous sequence of n items (words, characters) from text, used in language modeling and feature extraction. Language & LLMs Large Language Model A neural network, almost always a transformer, trained on vast amounts of text to predict the next token, which lets it write, answer, summarize and follow instructions. Language & LLMs Autoregressive Model A model that generates output one token at a time, using previously generated tokens as input for the next prediction.

In the research

All papers →

4 papers that build on Language Modeling .