C/C++ bindings for Huggingface Tokenizers.
- MLC tokenizers-cpp. We take inspiration from their work, and expand it to support more platforms & architectures.
muna-ai/libtokenizers
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
| Name | Name | Last commit date | ||
|---|---|---|---|---|
C/C++ bindings for Huggingface Tokenizers.