[User] Hitting special characters keys on AZERTY keyboards starts infinite generation in interactive mode

[BakaKiller]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

When I hit a "ù" or a "é" in interactive mode, the letter "ù" or "é" is written to the prompt.

Current Behavior

When I hit a "ù" or a "é", it ignores it and starts completting my prompt. Even a Ctrl+C doesn't stop it. I have to Ctrl+C several times for it to stop generating and close the program.

Environment and Context

Azerty keyboard

• Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   40 bits physical, 48 bits virtual
CPU(s):                          6
On-line CPU(s) list:             0-5
Thread(s) per core:              1
Core(s) per socket:              6
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           49
Model name:                      AMD EPYC 7282 16-Core Processor
Stepping:                        0
CPU MHz:                         2794.748
BogoMIPS:                        5589.49
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       384 KiB
L1i cache:                       384 KiB
L2 cache:                        3 MiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0-5
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_
                                 good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
                                  hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 
                                 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat umip rdpid arch_capabilities

• Operating System, e.g. for Linux:

$ uname -a

Linux vmixxxxxxx.contaboserver.net 5.10.0-12-amd64 #1 SMP Debian 5.10.103-1 (2022-03-07) x86_64 GNU/Linux

• SDK version, e.g. for Linux:

$ python3 --version
Python 3.9.2
$ make --version
GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ g++ --version
g++ (Debian 10.2.1-6) 10.2.1 20210110
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. Start llama.cpp in interactive mode and interactive first
2. hit an accentued character, for instance é

[FlorianJoncour]

Same issue if I set:

export LC_ALL=C

Which can be understood since stdin is in ASCII mode.

export LC_ALL=fr_FR.utf-8

Should solve your problem.
Seems not to be a llama.cpp issue btw.

[BakaKiller]

Thank you for your quick response.
Even if it solves the problem, why does it start a "non-stoppable completion" ? Can't it be considered as a bug ?
If not then I'll close the ticket

Edit : I could try it and it does work, so thanks for that too ^^

[staviq]

Same happens with polish or simply non-latin localized input.

I'm gonna try looking into this.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.