Ideas for pythainlp.lm function

[wannaphong]

I think pythainlp.lm class should collect the function for doing preprocessing or post-processing Thai text from LLM and include a small language model that can run in computers for home users to do simple NLP jobs.

Preprocessing

• pythainlp.lm.calculate_ngram_counts: Calculates the counts of n-grams in the list words for the specified range. Add pythainlp.lm.calculate_ngram_counts #1054

Post-processing

• pythainlp.lm.remove_repeated_ngrams: Remove repeated n-grams (to fixed lm) Add pythainlp.llm #1043

[bact]

If we're going to have a small language model as well, should we call the module just "lm"?
Just to make it more generic.

[wannaphong]

If we're going to have a small language model as well, should we call the module just "lm"? Just to make it more generic.

Agree 👍

[matichon-vultureprime]

How about leveraging NVIDIA-Curator to do pre-processing and post-processing?

We already have some examples from the NVIDIA team:

• Curating Non-English Datasets for LLM Training with NVIDIA NeMo Curator
• Training Localized Multilingual LLMs with NVIDIA NeMo

[wannaphong]

Add pythainlp.lm.calculate_ngram_counts #1054

[bact]

For the "small language model", what about having that model as a core/cores for most of the basic tasks that don't required larger model in PyThaiNLP? So we will have less dependencies as well.

Related to

• Porting model to ONNX model #639
• Porting Thai2fit from fastai v1 to fastai v2 #716
• Remove all python-crfsuite models from PyThaiNLP #655
• Consider reduce dependencies #935

[wannaphong]

For the "small language model", what about having that model as a core/cores for most of the basic tasks that don't required larger model in PyThaiNLP? So we will have less dependencies as well.

Related to

* [Porting model to ONNX model #639](https://github.com/PyThaiNLP/pythainlp/issues/639)

* [Porting Thai2fit from fastai v1 to fastai v2 #716](https://github.com/PyThaiNLP/pythainlp/issues/716)

* [Remove all python-crfsuite models from PyThaiNLP #655](https://github.com/PyThaiNLP/pythainlp/issues/655)

* [Consider reduce dependencies #935](https://github.com/PyThaiNLP/pythainlp/issues/935)

Just llama-cpp-python or onnx model. I think it is ok.