Build a BPE tokenizer - in your browser, from scratch
Every LLM bill you have ever paid started with a tokenizer. And if you work with Polish text, you are quietly paying about twice as much as your English-speaking friends - for the same sentence. Today we find out why, the best way I know: by building the thing.
Reading about tokenizers is like reading about swimming. So this post is the pool: a marimo notebook compiled to WebAssembly. Real Python runs in your browser tab - no install, no server, nothing you type leaves your machine. The whole lesson lives inside it:
- A naive regex tokenizer - and four real-world strings that break it.
- Byte Pair Encoding from scratch - three small functions, one of them yours to write (there is an exercise with a hidden solution).
- Train on Pan Tadeusz, test on the Constitution - watch Polish morphemes emerge from 1834 verse and compress 1997 legal text by ~1.8×.
- Crash a character-level tokenizer with an emoji - and see why byte-level shrugs it off. 🚀
- Measure the Polish token tax - the same sentence costs 32 tokens in Polish and 16 in English on GPT-4's tokenizer. Your API bills already know this.
- Four production trade-offs - special tokens, vocab size, tokenizer versioning, pretokenization.
- 🐍 Real Python (Pyodide/WASM) - runs locally in this tab
- 🏋️ One exercise, live inputs, a train button, zero setup
- 🔒 Nothing gets uploaded anywhere
Downloads ~20 MB of Python runtime on first launch - that is why it waits for your click. Prefer a separate tab? Open it full screen →
My favorite part, so you know what to look for: after training, check which tokens the algorithm invented. Nobody told it what a Polish word is - się, ość and prze emerge purely from pair statistics. Then it takes those poem-learned merges and compresses the Constitution. Transfer learning, 163 years across a genre gap, in 60 lines of Python you just wrote yourself.
What now?
If you enjoyed learning by poking a running artifact instead of nodding at slides - that is not an accident, that is the method:
Questions? Drop me a line at [email protected] or grab a 30-minute call here. Corpus credits: Pan Tadeusz via Wolne Lektury, Konstytucja RP via Wikisource - both public domain. See you on DemystifAI!