Skip to content

Unable to install azure-storage-blob dependencies via setup.py #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
SamuelRamond opened this issue Sep 5, 2018 · 5 comments
Closed
Labels
fixed/waiting for release describe issues that are fixed and sitting on the dev branch to be released later

Comments

@SamuelRamond
Copy link

Which service(blob, file, queue) does this issue concern?

>>> from azure.storage.blob import BlockBlobService
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named storage.blob

Which version of the SDK was used? Please provide the output of pip freeze.

azure-common==1.1.14
azure-nspkg==2.0.0
azure-storage-blob==1.3.1
azure-storage-common==1.3.0
azure-storage-nspkg==3.0.0

What problem was encountered?

Unable to install the whole dependency tree via classic setup.py in install_requires

Have you found a mitigation/solution?

Using pip install azure-storage-blob==1.3.1 it works well but in our CI process we try to have all dependency in our setup.py

I tried to add what seems to be the additional dependencies while pip freeze after manual pip install but this does not work either.
FYI, i added

    'azure-common==1.1.14',
    'azure-nspkg==2.0.0',
    'azure-storage-blob==1.3.1',
    'azure-storage-common==1.3.0',
    'azure-storage-nspkg==3.0.0',

Is there something i need to add in the setup.py to make it work ?

@zezha-msft
Copy link
Contributor

Hi @SamuelRamond, thanks for reaching out!

I was able to reproduce this issue by doing python azure-storage-blob/setup.py install. I'll look into how to fix this. Thanks!

@lmazuel
Copy link
Member

lmazuel commented Sep 19, 2018

So, the root cause is because setuptools is downloading the wheel of "azure-common", which he shouldn't do (setup.py install is supposed to not use wheel at all).

Searching for azure-common>=1.1.5
Reading https://pypi.python.org/simple/azure-common/
Downloading https://files.pythonhosted.org/packages/28/b9/51abe52b2d744e46f68314db31d178a8e8a2ea9a409b42146af08884ed47/azure_common-1.1.15-py2.py3-none-any.whl#sha256=8dda3d3eee6ec1b0d014aa84260042f6840d28d7b3fe03a552ed1e1541c00986
Best match: azure-common 1.1.15
Processing azure_common-1.1.15-py2.py3-none-any.whl
Installing azure_common-1.1.15-py2.py3-none-any.whl to c:\users\lmazuel\git\azure-storage-python\testenv\lib\site-packages
writing requirements to c:\users\lmazuel\git\azure-storage-python\testenv\lib\site-packages\azure_common-1.1.15-py3.7.egg\EGG-INFO\requires.txt
Adding azure-common 1.1.15 to easy-install.pth file

Installed c:\users\lmazuel\git\azure-storage-python\testenv\lib\site-packages\azure_common-1.1.15-py3.7.egg

I expect setuptools to download the sdist when installation is planned to be as egg. Our wheels and sdist are different, and if try a mixed installation of wheel and some egg, that has no chance to work out of the box.
I'm trying to understand why setuptools thinks it's a good idea to download a wheel and convert it to an egg...

Solution might be to downgrade setuptools. setup.py commands are considered deprecated, so I won't be surprised recent setuptools have unexpected result on setup.py. By the way, instead of install, you could use use pip install -e . which will install the current folder in editable mode with all dependencies, and use wheel files instead of eggs for them.

Edit: To support what I'm saying about deprecation:

@lmazuel
Copy link
Member

lmazuel commented Sep 19, 2018

So I took a random old setuptools (25.1.25) and retried it, and it uses the sdist as expected:

Searching for azure-common>=1.1.5
Reading https://pypi.python.org/simple/azure-common/
Downloading https://files.pythonhosted.org/packages/35/46/1cc91b5b6d3ea4ba239e202414102f196cd611f85afa77300432244b8902/azure-common-1.1.15.zip#sha256=8139e399d8c15f1dfcbc6254ccf98a835888cdaf500487b58d6af3e4bb3726a2
Best match: azure-common 1.1.15
Processing azure-common-1.1.15.zip
Writing C:\Users\lmazuel\AppData\Local\Temp\easy_install-afihut10\azure-common-1.1.15\setup.cfg
Running azure-common-1.1.15\setup.py -q bdist_egg --dist-dir C:\Users\lmazuel\AppData\Local\Temp\easy_install-afihut10\azure-common-1.1.15\egg-dist-tmp-et_yuz0w
Wheel is not available, disabling bdist_wheel hook
creating c:\users\lmazuel\git\azure-storage-python\testenv2\lib\site-packages\azure_common-1.1.15-py3.7.egg
Extracting azure_common-1.1.15-py3.7.egg to c:\users\lmazuel\git\azure-storage-python\testenv2\lib\site-packages
Adding azure-common 1.1.15 to easy-install.pth file

Installed c:\users\lmazuel\git\azure-storage-python\testenv2\lib\site-packages\azure_common-1.1.15-py3.7.egg

So, with pip and setuptools moving on, at some point setuptools team decided to download wheels to build egg file, which is surprising, but that's how it is...

Only solutions to this problem then:

  • Downgrade setuptools if you still want to do "python ./setup.py install". I won't suggest, since you will get in trouble with "cryptography" likely.
  • Use "pip install -e ." and recent pip/setuptools (recommended approach)
  • Uninstall azure-nspkg and azure-storage-nspkg after your "python ./setup.py install" (which are the root cause of your issue by the way, these packages are installed only by our wheel files and are messing up eggs installation).

@lmazuel
Copy link
Member

lmazuel commented Sep 19, 2018

Update: tracked this down to setuptools 38.2.0 (released in November 2017)
https://setuptools.readthedocs.io/en/latest/history.html#v38-2-0

easy_install now support installing from wheels: they will be installed as standalone unzipped eggs.

@zezha-msft zezha-msft added fixed/waiting for release describe issues that are fixed and sitting on the dev branch to be released later and removed investigating labels Oct 9, 2018
@zezha-msft
Copy link
Contributor

Fixed in 1.4.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fixed/waiting for release describe issues that are fixed and sitting on the dev branch to be released later
Projects
None yet
Development

No branches or pull requests

3 participants