Update newmm.py

wannaphong · wannaphong · commit 3d889f7be4cf · 2023-11-01T19:57:07.000+07:00
diff --git a/pythainlp/tokenize/newmm.py b/pythainlp/tokenize/newmm.py
@@ -44,7 +44,7 @@
 \d+([,\.]\d+)*|    # numbers
 [ \t]+|            # spaces
 \r?\n|             # newlines
-[^\u0E00-\u0E7F \t]+  # other non-Thai characters
+[^\u0E00-\u0E7F \t\r\n]+  # other non-Thai characters
 """
 )
 

Original file line number	Diff line number	Diff line change
`@@ -44,7 +44,7 @@`
`44`	`44`	`\d+([,\.]\d+)*\| # numbers`
`45`	`45`	`[ \t]+\| # spaces`
`46`	`46`	`\r?\n\| # newlines`
`47`		`-[^\u0E00-\u0E7F \t]+ # other non-Thai characters`
	`47`	`+[^\u0E00-\u0E7F \t\r\n]+ # other non-Thai characters`
`48`	`48`	`"""`
`49`	`49`	`)`
`50`	`50`