Sentence Tokenizer Split sentence on Terminal Puntuation.

[varunkatiyar819]

Detailed description

Currently the Crfcut engine for the sentence tokenize part does not split the sentence correctly for terminal punctuations, for enders it is working fine, but when it comes to split using terminal punctuation commonly '.' (full stop), it's still creating a issue. I tried with different engines too, but i could not get the desired output for the sentence tokenization. Since also came across it is trained on TED Dataset so it may certainly have not came across that maybe.

Context

Trying to look for this fix to handle formatting of the sentence after translation.
Also, would be useful for those looking for similar use case.

Possible implementation

This is what currently i was tring for my use case. Below is the code provided :
for idx, _ in enumerate(toks):
if(toks[idx].strip().endswith(('!', '.', '?'))):
labs[idx] = "E"

    #Spaces would no longer would be treated as end of the sentences.
    elif(toks[idx].strip() == ""):
        labs[idx] = "I"

Your environment

• PyThaiNLP version: 3.1.1
• Python version: 3.10
• Operating system and version (distro, 32/64-bit): 64 bit
• More info (Docker, VM, etc.):

[github-actions]

Hello @varunkatiyar819, thank you for your interest in our work!

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

[varunkatiyar819]

Here i have attached all the cases where, crfcut fails to split using terminal punctuations in some cases. But since terminal punctuations marks the end of sentence, it should always split, but many a cases it is not working, current fix to it i have added is given below :

def segment(text: str) -> List[str]:
"""
CRF-based sentence segmentation.

    :param str text: text to be tokenized to sentences
    :return: list of words, tokenized from the text
    """
    if isinstance(text, str):
        toks = word_tokenize(text)
    else:
        toks = text
    feat = extract_features(toks)
    labs = _tagger.tag(feat)
    labs[-1] = "E"  # make sure it cuts the last sentence
**
    for idx, _ in enumerate(toks):
          if(toks[idx].strip().endswith(('!', '.', '?'))):
              labs[idx] = "E"
        
        #Spaces would no longer would be treated as end of the sentences.
        elif(toks[idx].strip() == ""):
            labs[idx] = "I"
**

    sentences = []
    sentence = ""

    for i, w in enumerate(toks):
        sentence = sentence + w
        **if labs[i] == "E" and sentence != '':**
            sentences.append(sentence)
            sentence = ""

    return sentences