python arm build issue still failing in some cases

[blathersbugs]

Description:

python lambda function builds using the sam cli (which calls this library) are still failing for certain dependencies

Steps to reproduce the issue:

1. setup python project with sam cli
2. add py7zr to the requirements.txt of a function
3. run command sam build -u

Observed result:
build fails on resolve dependencies
Expected result:
build completes
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
aws codebuild
image
aws/codebuild/amazonlinux2-x86_64-standard:3.0
buildspec.yaml:
version: 0.2
env:
secrets-manager:
DOCKER_PASSWORD: *******
DOCKER_USER: ******
phases:
install:
runtime-versions:
python: 3.9
build:
commands:
- printenv
- pip install --upgrade aws-sam-cli
- docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
- echo $DOCKER_PASSWORD | docker login --username $DOCKER_USER --password-stdin
- sam build -u --debug
- sam package --s3-bucket ******i --output-template-file template-out.yaml
artifacts:
files:
- template-out.yaml

sam build -u --debug output near failure:

2022-03-22 18:35:43,680 | Building codeuri: /codebuild/output/src531839338/src/lambda/******** runtime: python3.9 metadata: {} architecture: arm64 functions: ['*****']
2022-03-22 18:35:43,680 | Building to following folder /codebuild/output/src531839338/src/.aws-sam/build****

Fetching public.ecr.aws/sam/build-python3.9:latest-arm64 Docker container image......
2022-03-22 18:35:44,617 | Mounting /codebuild/output/src531839338/src/lambda/****** as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Registering workflow 'NodejsNpmEsbuildBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm-esbuild', application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(sdist), texttable==1.6.4(wheel), zipfile-deflate64==0.0.0(sdist), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
initial compatible: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
initial incompatible: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
Downloading missing wheels: {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch pycryptodomex==3.14.1
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch zipfile-deflate64==0.0.0
compatible wheels after second download pass: {brotli==1.0.9(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel)}
Build missing wheels from sdists (C compiling True): {pycryptodomex==3.14.1(sdist), zipfile-deflate64==0.0.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/pycryptodomex-3.14.1.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz
compatible after building wheels (no C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Build missing wheels from sdists (C compiling False): {zipfile-deflate64==0.0.0(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/zipfile-deflate64-0.2.0.tar.gz
compatible after building wheels (C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Final compatible: {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
Final incompatible: set()
Final missing wheels: {zipfile-deflate64==0.0.0(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
package_builder.build_dependencies(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run
action.execute()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}
Builder workflow failed
Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
package_builder.build_dependencies(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 156, in build_dependencies
self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 302, in run
action.execute()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 57, in execute
raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {zipfile-deflate64==0.0.0(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/main.py", line 118, in main
builder.build(
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/builder.py", line 164, in build
return workflow.run()
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 96, in wrapper
func(self, *args, **kwargs)
File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflow.py", line 309, in run
raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}
2022-03-22 18:43:02,875 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}"}}

Build Failed
2022-03-22 18:43:03,179 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '64785e7d-a17c-4f7f-a347-aca7b7adc804', 'installationId': '75a7108d-f9a0-4aa3-ad9a-b637b8ea0647', 'sessionId': 'c7ec421f-a00b-47f1-b1e6-b34189e99768', 'executionEnvironment': 'AWSCodeBuild', 'ci': True, 'pyversion': '3.9.5', 'samcliVersion': '1.41.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'metricSpecificAttributes': {'projectType': 'CFN'}, 'duration': 667631, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
2022-03-22 18:43:03,217 | Telemetry response: 200
Error: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}

[Container] 2022/03/22 18:43:03 Command did not exit successfully sam build -u --debug exit status 1
[Container] 2022/03/22 18:43:03 Phase complete: BUILD State: FAILED
[Container] 2022/03/22 18:43:03 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: sam build -u --debug. Reason: exit status 1
[Container] 2022/03/22 18:43:03 Entering phase POST_BUILD
[Container] 2022/03/22 18:43:03 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2022/03/22 18:43:03 Phase context status code: Message:
[Container] 2022/03/22 18:43:03 Expanding base directory path: .
[Container] 2022/03/22 18:43:03 Assembling file list
[Container] 2022/03/22 18:43:03 Expanding .
[Container] 2022/03/22 18:43:03 Expanding file paths for base directory .
[Container] 2022/03/22 18:43:03 Assembling file list
[Container] 2022/03/22 18:43:03 Expanding template-out.yaml
[Container] 2022/03/22 18:43:03 Skipping invalid file path template-out.yaml
[Container] 2022/03/22 18:43:03 Phase complete: UPLOAD_ARTIFACTS State: FAILED
[Container] 2022/03/22 18:43:03 Phase context status code: CLIENT_ERROR Message: no matching artifact paths found

[mildaniel]

Thanks for reporting the issue, we will be investigating it further. Please watch this channel for more updates, and feel free to reach out.

[blathersbugs]

another function with the following requirements:

requests
python_jwt

plus a layer with this:

dynamodb-encryption-sdk

that was failing to build before is now building successfully, if that helps.
(the layer was always succeeding)
none of the versions is pinned, so unfortunately that makes more variables to deal with.

[mndeveci]

@blathersbugs thanks for raising this issue.

I've tried with following dependencies with python3.9 runtime, and I confirm that it works if I run sam build -u;

requests
python_jwt
dynamodb-encryption-sdk

Can you provide more details for these ones (python version for instance) to test it further?

I also tried with py7zr as dependency, but that one fails with following message;

Build Failed
Running PythonPipBuilder:ResolveDependencies
Error: PythonPipBuilder:ResolveDependencies - {zipfile-deflate64==0.0.0(sdist)}

When I check PyPI, I can't find the 0.0.0 version of zipfile-deflate64. Do you know where that dependency & version comes from?

[blathersbugs]

hi @mndeveci

requests
python_jwt
dynamodb-encryption-sdk

work as expected, so there is nothing to reproduce there.
I don't know where where it gets the 0.0.0 version.
If you look through the output, you can see it builds the correct wheel here
compatible after building wheels (C compiling): {brotli==1.0.9(wheel), pycryptodomex==3.14.1(wheel), texttable==1.6.4(wheel), pybcj==0.5.0(wheel), multivolumefile==0.2.3(wheel), pyppmd==0.17.4(wheel), pyzstd==0.15.2(wheel), py7zr==0.18.1(wheel), zipfile-deflate64==0.2.0(wheel)}
but because of the bogus 0.0.0 version, it believes it does not have all dependencies and the build fails.
I was using python 3.9.

if you download the py7zr source files it lists the following as its depedencies:
dependencies = [ "texttable", "pycryptodomex>=3.6.6", 'importlib_metadata;python_version<"3.8"', 'brotli>=1.0.9;platform_python_implementation=="CPython"', 'brotlicffi>=1.0.9.2;platform_python_implementation=="PyPy"', "pyzstd>=0.14.4", "pyppmd>=0.18.1,<0.19.0", 'pybcj>=0.5.0;platform_python_implementation=="CPython"', "multivolumefile>=0.2.3", "zipfile-deflate64>=0.2.0", ]
so I would say there is an issue in the dependency resolution

[blathersbugs]

hi @mndeveci @mildaniel any updates?
is there any more information you need from me?

[mndeveci]

Hi @blathersbugs ,

I've retired with following function definition;

 HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - arm64

with following requirements.txt file;

py7zr
requests
python_jwt
dynamodb-encryption-sdk

when I run sam build -u I got my function built successfully.

I remember this was failing before so please let me know if I am missing something here. Otherwise I would assume the library have fixed the issue with their dependencies that we discussed earlier.

Thanks