publish v2 to PyPI

[mattsb42-aws]

It's great that there's a first-class supported installation mechanism that does not rely on environment specifics, but this is still a normal Python package, and for those customers who do already have an existing compatible environment the installation mechanism is a lot heavier-weight than using the existing Python distribution mechanisms.

Could we get v2 publishing to PyPI so that those customers who have those prerequisites can have a simpler and easier to maintain installation/updating/etc story?

[joguSD]

Not publishing CLI v2 to PyPI has been a conscious decision and for the time being we have no plans to publish to PyPI. In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature.

[sbbowers]

@joguSD Could you please elaborate on the reasoning behind this conscious decision? Perhaps link us to some dialog around this discussion?

[ssbarnea]

I think that time may prove if that decision was more towards conscious or in-conscious one. I am really curious to see how this will evolve, I only hope I have enough popcorn 🍿.

On the other hand, other wheels coming from the same source did not get their notoriety without any reasons. boto, boto3 anyone?

At least pypi was 3rd party that guaranteed that a specific version is not altered once uploaded. I guess now we will have to keep saving checksums .

[mattsb42-aws]

So pip install git+git://github.com/aws/aws-cli.git#2.0.0 then. :)

[ssbarnea]

Sure, we all know that is impossible to force push code to github, not.

[adamchainz]

Or use a tarballs link for greater speed https://adamj.eu/tech/2019/03/11/pip-install-from-a-git-repository/

[joguSD]

Just as a reminder, the AWS CLI V1 will continue to receive regular updates
that are published to PyPI and pip will continue to be a supported
installation mechanism for AWS CLI V1.

Here are some reasons why we dropped publishing to PyPI for AWS CLI V2, and are
not supporting pip as an official installation mechanism.

1. Installing with pip requires Python knowledge. The AWS CLI isn't a tool
   exclusive to Python developers and it's very easy to end up with an invalid
   installation. Often, reported issues end up being due to a misconfigured pip
   install that's getting dependencies from the wrong place, etc. These issues
   happen unexpectedly and are hard to debug which is frustrating for customers.
   It's also possible to inadvertently break other packages in the same
   environment.

2. Depending on a C extension can mean that at pip install time a C compiler
   and any other required libraries may need to be present for the install to
   succeed. Because of this we've avoided depending on libraries that require C
   extensions as having a compiler wasn't previously a requirement to install the
   AWS CLI. We also do not feel comfortable adding a compiler as a requirement to
   install the AWS CLI V2. For V1 this has limited our ability to implement some features
   as there are no suitable Python native libraries that meet our requirements
   (cryptography, etc).

3. Supporting pip as an installation mechanism requires supporting large
   dependency ranges for the packages we depend on. While simple in concept, this
   is difficult to execute on reliably as it's not feasible to test all
   permutations of platform, Python version, and package version. When new
   packages we depend on are published V1 customers installing via pip are often
   broken. This leads us to have relatively aggressive version ranges and we end
   up becoming a blocker for the Python community to upgrade packages.
   See upgrade to PyYAML 5.2 #4749 PIP dependency conflict on colorama with AWSCLI #4561 Installation is broken: Requirement.parse('urllib3<1.23,>=1.21.1') #3535

4. There are some things we have no control of when installing via pip. For
   example, we have no influence on the openssl version or what TLS related
   features are supported as those will depend on how Python was built. This makes
   it difficult to ensure functionality of the AWS CLI.

5. We cannot include or ship additional binaries (that may not even be written
   in Python) as part of an AWS CLI install. In the future the AWS CLI can be
   partially or even entirely rewritten in another language while maintaining
   backwards compatibility.

Hopefully that helps to shed some light on why we decided not to publish the
AWS CLI V2 to PyPI.

Lastly, we're looking to continue to improve the installation process for AWS
CLI V2 by potentially integrating with homebrew and Linux package managers, and
providing docker images. We'll also be looking to include build instructions
for cases where the pre-built binaries aren't available or suitable for a
particular use case.

[ulidtko]

Those sound as decent and understandable reasons — except perhaps the #1. I'd avoid going into language comparison contests here — but it's hard to refute that Python is among the few must know languages for DevOps folks and the like.

Likely, I'm being ignorant in this judgement; but one can be forgiven for not "really knowing" even Bash — but not Python.

It's the language of humanity's first ever photo of a [shadow of] black hole, remember?

---

Besides: that's some really wierd reasoning, don't you think?

• Some users have hard time with pip, while others (most?..) pip users are quite happy and silent.
• Yet others have hard time with prebuilt binaries, while others (most?..) binariy users are quite happy and silent.
• Therefore, we'd punish ALL pip users, including the happy&silent ones, and force them to go use binaries.

Edit: sorry, disregard this.

---

Lastly: pip install https://github.com/aws/aws-cli/archive/2.0.0.tar.gz fails with:

No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0)

What's the route to go to install aws-cli v2 from source?

[quiver]

@ulidtko

Not only aws-cli but also botocore(low-level API) is not published to PyPI for v2, so you need to install botocore beforehand.

For example, on a vanilla A1(ARM-base) Amazon Linux 2 instance, you can install AWS-CLI v2 as follows:

$ sudo yum install -y python3 python3-devel gcc libffi-devel openssl-devel
$ python3 -mpip install  https://github.com/boto/botocore/archive/v2.tar.gz
$ python3 -mpip install  https://github.com/aws/aws-cli/archive/v2.tar.gz
$ aws --version
aws-cli/2.0.0 Python/3.7.4 Linux/4.14.165-131.185.amzn2.aarch64 botocore/2.0.0dev4

[thimslugga]

What about providing an AppImage for Linux vs the zip archive with the installer? I created one using the contents from the dist dir and it seems to be working fine. Just throwing it out there as it may be worthwhile way to distribute the v2 release for Linux. I can share steps I used if there's any interest.

[bristea]

That like is like gold

No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0)

Translates to "We made a release using unreleased dependencies." and "We didn't test if the release artifact can be installed".

Nobody mentioned is that binary code downloaded needs to signed on MacOS, so which means that the so called brew alternative install method will take considerable amount of effort/time.

Let's face it, v2 was released even without a tag being made of github, this should say a lot. AFAIK it could also be made from another repository, another license. The v2 was not presented as a preview and was directly published on https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html as a recommended release when the code repository, didn't had any tag, prerelease or not.

I would think twice before arguing that so many users are not able to use pip, or compile code, when basic software development practices like tagging are not followed.

[sbbowers]

Installing with pip requires Python knowledge...

I agree. Doing things in a domain requires domain knowledge.

Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present...

This is more domain knowledge, sure.

Supporting pip as an installation mechanism requires supporting...

It seems to me that this is your crux. Supporting PIP costs time and money and it sounds like it is generally painful. I suspect this is because, of the people that need support, they don't have the domain knowledge you'd mentioned.

There is a difference between "supporting PIP as the method of install" vs "uploading your package to PyPi". You can keep on delivering your curl/unzip/bash method of deployment as official and supported and also allow, for the people that know and want to, install v2 via pip as an "unsupported" option.

There are some things we have no control of when installing via pip...

You may not have control over it, but you can check for it and make it dependent on a specific version.

We cannot include or ship additional binaries...

Additional binaries... sounds like dependencies, which could be checked for during the build process.

The take-away I'm trying to make here is that supplying your packages via PIP isn't antithetical to your desire to limit support to your official installer. Would you please reconsider your aversion to pypi? Many developers would appreciate it.

[ulidtko]

There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.

Don't you think that this is quite intentional?..

To a user of your package, that's an atrocious idea: that installing an API client tool gives control over openssl on my system to the tool author. I'm getting shivers just from thinking of it!

Constructively, there're two things you can do:

• outright fail pip build, if the given openssl isn't good enough;
• graceful degradation, where parts of functionality which depend on missing TLS features just get disabled.

---

We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.

Why, what's wrong with setup(data_files=...) and/or MANIFEST.in, did you look into it?

Thousands of developers do include arbitrary files in their pip installs. I see no problem here.

Oh wait, actually I do. Binaries don't belong in source installs.

If I'd want to rewrite aws-cli in golang or something — I'd make it separate project, in a separate repo, and maintain its interface as "CLI compatible" rather than vaguely "backwards-compatible".

Don't ship binaries where source is expected. No-no. No no no no.