Skip to content

Commit 087566d

Browse files
committed
Sync llama-vocab : add SuperBPE pre-tokenizer
1 parent 94f4358 commit 087566d

File tree

1 file changed

+2
-0
lines changed

1 file changed

+2
-0
lines changed

llama_cpp/llama_cpp.py

+2
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,7 @@
238238
# LLAMA_VOCAB_PRE_TYPE_MINERVA = 27,
239239
# LLAMA_VOCAB_PRE_TYPE_DEEPSEEK3_LLM = 28,
240240
# LLAMA_VOCAB_PRE_TYPE_GPT4O = 29,
241+
# LLAMA_VOCAB_PRE_TYPE_SUPERBPE = 30,
241242
# };
242243
LLAMA_VOCAB_PRE_TYPE_DEFAULT = 0
243244
LLAMA_VOCAB_PRE_TYPE_LLAMA3 = 1
@@ -269,6 +270,7 @@
269270
LLAMA_VOCAB_PRE_TYPE_MINERVA = 27
270271
LLAMA_VOCAB_PRE_TYPE_DEEPSEEK3_LLM = 28
271272
LLAMA_VOCAB_PRE_TYPE_GPT4O = 29
273+
LLAMA_VOCAB_PRE_TYPE_SUPERBPE = 30
272274

273275

274276
# // note: these values should be synchronized with ggml_rope

0 commit comments

Comments
 (0)