Skip to content

Commit 6fcbf68

Browse files
llama : implement Unigram tokenizer needed by T5 and FLAN-T5 model families (#5763)
* llama : add T5 model architecture, tensors and model header parameters * llama : add implementation of Unigram tokenizer with SentencePiece-like text normalization using precompiled charsmap --------- Co-authored-by: Stanisław Szymczyk <[email protected]>
1 parent e6bf007 commit 6fcbf68

File tree

4 files changed

+587
-39
lines changed

4 files changed

+587
-39
lines changed

0 commit comments

Comments
 (0)