Skip to content

Commit 448f398

Browse files
committed
potential out of bounds read
1 parent 5ae5e2c commit 448f398

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

utils.cpp

+1-1
Original file line numberDiff line numberDiff line change
@@ -323,7 +323,7 @@ struct llama_tokenizer {
323323
int index = 0;
324324
while (!text.empty()) {
325325
llama_sp_symbol sym;
326-
size_t char_len = utf8_len(text.data()[0]);
326+
size_t char_len = std::min(text.size(), utf8_len(text.data()[0]));
327327
sym.text = std::string_view(text.data(), char_len);
328328
sym.prev = index - 1;
329329
text.remove_prefix(char_len);

0 commit comments

Comments
 (0)