Add: remove_trailing_repeat_consonants()

[konbraphat51]

What does this changes

Add removement method of repeating consonants by dictionary-based method.

How this fixes it

Add pythainlp.utils.remove_repeat_consonant(text, dictionary)

Find words in the dictionary that has repeating consonant at the last, and find the match with the end of the sentence. If there is none, make the repetition to one; if there is, make the repetition to that one.

Fixes #860

Your checklist for this pull request

🚨Please review the guidelines for contributing to this repository.

• Passed code styles and structures
• Passed code linting checks and unit test

[pep8speaks]

Hello @konbraphat51! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file pythainlp/util/remove_trailing_repeat_consonants.py:

Line 51:69: W291 trailing whitespace
Line 126:9: W503 line break before binary operator
Line 128:9: W503 line break before binary operator
Line 249:13: W503 line break before binary operator
Line 249:50: E203 whitespace before ':'
Line 251:13: W503 line break before binary operator

Comment last updated at 2023-11-13 07:50:03 UTC

[konbraphat51]

Hmm... I cannot find more PEP8 error,,,
I will try more next morning. Good night!

[coveralls]

coverage: 86.937% (+86.9%) from 0.0%
when pulling 3315cb0 on konbraphat51:dev
into edb52b3 on PyThaiNLP:dev.

[konbraphat51]

Lint issue of normalize.py solved

[konbraphat51]

Okey, Lint problem and CodeClimate problem are basically solved, and the test for this passed.
Review please!

It seems that the code length of a single file is set to 250 lines, but should I separate this method from normalize.py?

[konbraphat51]

• Files seperated
• remove_trailing_repeat_consonants() sounds good to me, so I changed the name to that.
• I think I clarified names in the script, but if there are still name inconsistencies, please specify them.
• I revert normalize.py to what it was, so the Lint is pointing it out now
• Not sure why codeacy saying "pythainlp.util.remove_trailing_repeat_consonants.remove_trailing_repeat_consonants' imported but unused." It is used in test code. Any idea?

[bact]

I think I clarified names in the script, but if there are still name inconsistencies, please specify them.

Sorry. I was meant to say that the existing remove_repeat_vowels() and remove_dup_spaces() have naming inconsistency issue. Not from the new function you're now contributing.

[bact]

Neat contribution, thank you. I think that's all. I can approve after few suggested fixes above.

[konbraphat51]

Ok, all done.
Review please :)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[bact]

I have a last minute thought about which name we should go with,
between:

• remove_trailing_repeat_xxx vs
• remove_repeat_trailing_xxx

(try to think what it will like in the context of code completion, when ppl type remove_ .. which char they)

But I will merge this first.
We can later thinking more about consistent names (together with that remove_dup_xxx) and change this again when appropriate.

Thank you for your contribution.

(the test on Windows is now running for 58 minutes and counting, this shouldn't be the norm :( )