"nercut" result not printed in a specific sentence

[kmining]

Description

I try to tokenize text with ทุ๊กกโคนน , อือหือ, อย่าลืมอัพการ์ดนะจ๊ะ using nercut tokenizer.

Expected results

Something should be printed..

Current results

Result not printed

Steps to reproduce

import pythainlp
from pythainlp import word_tokenize
word_tokenize("ทุ๊กกโคนน", engine="nercut")

Your environment

• PyThaiNLP version: 3.0.5
• Python version: 3.7.3
• Operating system and version (distro, 32/64-bit): 64
• More info (Docker, VM, etc.):

[github-actions]

Hello @kmining, thank you for your interest in our work!

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

[wannaphong]

@kmining Thank you for reporting. I was fix this issue and I will release PyThaiNLP 3.0.6 to fix.

[kmining]

@wannaphong, Thanks for the quick response.
The error with no result has been resolved, but it seems that there is a duplicate of the extracted tokens.
In version 3.0.5, there were no duplicate tokens.

import pythainlp
from pythainlp import word_tokenize
word_tokenize("ทันแน่ๆ", engine="nercut")
word_tokenize("%1ครั้ง", engine="nercut")

[wannaphong]

@wannaphong, Thanks for the quick response. The error with no result has been resolved, but it seems that there is a duplicate of the extracted tokens. In version 3.0.5, there were no duplicate tokens.

import pythainlp
from pythainlp import word_tokenize
word_tokenize("ทันแน่ๆ", engine="nercut")
word_tokenize("%1ครั้ง", engine="nercut")

Thank you for report. I was rewrite code, so if it be done, I will release PyThaiNLP v3.0.8 to fix this issue.

[wannaphong]

@kmining It's done. Thank you for reporting.