tok (Q5983037)
From MaRDI portal
 | This is the item page for this Wikibase entity, intended for internal use and editing purposes. Please use this page instead for the normal view: tok |
Fast Text Tokenization
| Language | Label | Description | Also known as |
|---|---|---|---|
| English | tok |
Fast Text Tokenization |
Statements
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
0 references
17 August 2023
0 references
