Reference 3 stops to get here

WordPiece

A subword tokenization algorithm used by BERT, similar to BPE but with different merging criteria.

Your route here

3 stops · basics first
  1. Token ✓ understood

    The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.

  2. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  3. Subword Tokenization ✓ understood

    Breaking words into smaller units, balancing vocabulary size with representation granularity.

  4. WordPiece · you are here ✓ understood

A subword tokenization algorithm used by BERT, similar to BPE but with different merging criteria.

This concept is essential for understanding large language models and forms a key part of modern AI systems.

  • Tokenization
  • BPE
  • BERT

Where it sits

Before this

Subword Tokenization
WordPiece

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Language & LLMs BPE Byte Pair Encoding - a subword tokenization algorithm that iteratively merges frequent character pairs to create a vocabulary. Language & LLMs SentencePiece A language-agnostic tokenization library that treats text as a sequence of Unicode characters. Language & LLMs BERT Bidirectional Encoder Representations from Transformers - a model that understands context by looking at text from both directions.