Atlas · GenAI 2026
Tokenization
Splitting text into subword units for models (BPE, WordPiece, SentencePiece); underlies context limits, cost and multilingual behavior.
conceptPeak: 2020NLP FoundationsAI consensus: 0/3
Prerequisites
- mediumNLP
Tokenization is the first step of the NLP pipeline.
Recommended reference
Neural Machine Translation of Rare Words with Subword Units — arXiv 1508.07909