pip install would failed when there is something wrong in ～\.python_history

[eromoe]

Environment

• pip version: 19.03
• Python version: 3.7.3
• OS: win10 x64

Today I updated anaconda, then pip install -r requirements.txt got error

Exception:
Traceback (most recent call last):
  File "C:\Anaconda3\envs\py3\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
    status = self.run(options, args)
  File "C:\Anaconda3\envs\py3\lib\site-packages\pip\_internal\commands\install.py", line 289, in run
    self.name, wheel_cache
  File "C:\Anaconda3\envs\py3\lib\site-packages\pip\_internal\cli\base_command.py", line 289, in populate_requirement_set
    use_pep517=options.use_pep517):
  File "C:\Anaconda3\envs\py3\lib\site-packages\pip\_internal\req\req_file.py", line 103, in parse_requirements
    filename, comes_from=comes_from, session=session
  File "C:\Anaconda3\envs\py3\lib\site-packages\pip\_internal\download.py", line 445, in get_file_content
    content = auto_decode(f.read())
  File "C:\Anaconda3\envs\py3\lib\site-packages\pip\_internal\utils\encoding.py", line 38, in auto_decode
    locale.getpreferredencoding(False) or sys.getdefaultencoding(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0xab in position 312: illegal multibyte sequence

Only after I removed all unicode comments from requirements.txt , it became normal .

I also found enter python interpreter would print

Type "help", "copyright", "credits" or "license" for more information.
Failed calling sys.__interactivehook__
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site.py", line 439, in register_readline
    readline.read_history_file(history)
  File "C:\Anaconda3\lib\site-packages\pyreadline\rlmain.py", line 165, in read_history_file
    self.mode._history.read_history_file(filename)
  File "C:\Anaconda3\lib\site-packages\pyreadline\lineeditor\history.py", line 82, in read_history_file
    for line in open(filename, 'r'):
UnicodeDecodeError: 'gbk' codec can't decode byte 0xb1 in position 1182: illegal multibyte sequence

After some effort I found remove ～\.python_history solve this problem .

[chrahunt]

Unless the file has a BOM or e.g. # -*- coding: utf-8 -*- then we treat the file according to the the preferred locale or default encoding. I'm not sure we can do any better here. You can check the output of python -c "import locale, sys; print(f'{locale.getpreferredencoding(False)}:{sys.getdefaultencoding()}')" to determine what encoding we would've assumed for the text.

We may document something about the fact that we support coding: ***, for people who want to use a specific encoding but also want it to be processed in a platform/configuration-independent way.

The ~\.python_history issue seems unrelated to pip to me.

[eromoe]

I use utf-8 everywhere and didn't modify this file encoding at that time, but it was contaminated .
At least, error message should change to Please check your .python_history encoding, its path: xxxx

[chrahunt]

Here is my understanding, please tell me if I get something wrong:

There are two issues.

Issue 1: Running pip install -r requirements.txt fails with the first error text in the original issue above. To fix the issue, you removed lines that contain UTF-8 in requirements.txt. After the lines were removed, pip install -r requirements.txt succeeded.

Issue 2: Running python fails with the second error text in the original issue above. To fix the issue you removed ~\.python_history.

I think that both issue 1 and issue 2 are caused by the system encoding being set to "gbk" but the files containing unicode encoded in UTF-8.

---

Based on that understanding, my opinion is that:

1. We can make issue 1 less likely to hurt people by documenting that you can put # -*- coding: utf-8 -*- in your requirements.txt if you want it to always be treated like it has utf-8. This should already work in the current version of pip and you can put it in your requirements.txt if you want.
2. Issue 2 isn't a pip issue since only python is involved, so I don't think there's anything that we should do related to this. It looks like it is already tracked by Incorrect encoding of history files pyreadline/pyreadline#30 as mentioned in bpo-24811. I would go 👍 that issue since it impacts you.

[chrahunt]

Hello! I created an issue to track documenting that "coding: utf-8" is supported in requirements files, so I will close this one.