Rust corePython API

Exact BPE.
Corpus scale.

Fast and faithful byte-pair encoding for large, multilingual corpora—without approximating the model.

Start with Python Explore Rust API
pip install ffbpe
corpus.ffbpe exact

Input stream

你好世界 ·tokenizer
m_0842 [你][好] 48,392
m_0843 [token][izer] 31,705
cutoff freq ≥ measured boundary exact
606K unique words
3.70s BPE training

One release run · 64 MiB Chinese fixture · not a universal speed claim.

Python 3.11+ Native Rust core MIT licensed tiktoken-shaped API GPT-2 + unitoken formats

Designed for the hard part

Keep the model exact.
Change the constraints.

FFBPE separates corpus shaping, counting, training, and encoding so each stage can scale without quietly changing the learned model.

01

Shape Unicode-heavy inventories

Retain frequent adjacent Unicode pairs and split unproductive boundaries before BPE training. The measured cutoff travels with the inventory.

02

Bound memory, not correctness

An optional hot-pair window bounds persistent occurrence postings while preserving global frequencies and deterministic winners.

03

Stream through native batches

Feed replayable Python iterables into bounded Rust batches, merge partitioned counters, and avoid a corpus-sized Python dictionary.

04

Encode with familiar contracts

Save self-describing models, use GPT-2 or lossless Unicode files, and bring existing integrations through a tiktoken-shaped API.

ffbpe.json vocab.json merges.txt

Five-minute path

Train. Validate.
Ship the model.

The high-level Python API covers the common path. Drop down to explicit counters and trainers only when the corpus demands it.

Read the Python quickstart
quickstart.py 
from ffbpe import BpeEncoder, train_bpe

model = train_bpe(
  ["hello world", "hello tokenizer"],
  vocab_size=280,
  special_tokens=["<|endoftext|>"],
)

ids = model.encode("hello world")
assert model.decode(ids) == "hello world"

model.save_pretrained("my-tokenizer")
encoder = BpeEncoder.from_pretrained("my-tokenizer")

Scroll horizontally to see the complete example

Measured, qualified, reproducible

A smaller inventory.
A shorter training loop.

One release run · FineWeb2 Chinese fixture 64 MiB / 10K vocabulary
Regular Unicode 1,803,009 words
26.681s
Retained bigrams 606,153 words
3.702s

This comparison measures FFBPE inventory shaping. Segmentation changes, so it is not a model-parity claim. Always benchmark representative text.

Read the benchmark contract

Start exact

Your corpus is the benchmark.

Read the docs View on GitHub