tok3niz3r

A byte-level BPE tokenizer for Rust. Train a vocabulary from text, then encode and decode with it. It reads and writes HuggingFace tokenizer.json files, and it also has its own binary format that loads without parsing JSON and can be memory-mapped.

The project is split into separate crates so you depend only on the part you need.

Quick start

Command-line tool:

cargo install tok3niz3r-cli
tok3 encode --tokenizer tokenizer.json --text "Hello, world!"

Library (encode and decode an existing tokenizer):

[dependencies]
tok3niz3r-runtime = { version = "0.1", features = ["hf-compat"] }

Crates

Crate	Use it when you want to
tok3niz3r-cli	Do the above from the command line (the tok3 binary).
tok3niz3r-runtime	Load a tokenizer and encode/decode text.
tok3niz3r-trainer	Train a tokenizer from a corpus.
tok3niz3r-core	Build your own tooling on the shared types. Usually a transitive dependency only.

Each crate has its own README with a short example, and full API docs are on docs.rs.

License

MIT. See LICENSE.md.