Incorrect CMake module paths since aws-crt-cpp dependency was added

[glaubitz]

In bb96468, CMakeLists.txt was modified to use non-standard CMake modules paths which is incompatible with most standard distributions.

Trying to configure aws-sdk-cpp on openSUSE/SLE therefore currently fails with:

glaubitz@suse-laptop:~/upstream/aws-sdk-cpp/build> cmake -G Ninja ..
-- TARGET_ARCH not specified; inferring host OS to be platform compilation target
-- Building AWS libraries as shared objects
-- Generating linux build config
-- Building project version: 1.9.21
-- The CXX compiler identification is GNU 10.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11") 
--   Zlib library: /usr/lib64/libz.so
-- Encryption: Openssl
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.1.1k")  
--   Openssl include directory: /usr/include
--   Openssl library: /usr/lib64/libssl.so;/usr/lib64/libcrypto.so
-- Http client: Curl
-- Found CURL: /usr/lib64/libcurl.so (found version "7.76.1")  
--   Curl include directory: /usr/include
--   Curl library: /usr/lib64/libcurl.so
-- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC
-- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC - Success
CMake Error at CMakeLists.txt:174 (include):
  include could not find requested file:

    AwsFindPackage


CMake Error at CMakeLists.txt:184 (add_subdirectory):
  The source directory

    /home/glaubitz/upstream/aws-sdk-cpp/crt/aws-crt-cpp

  does not contain a CMakeLists.txt file.


-- Add s3-crt:s3 to C2J_SPECIAL_NAME_LIST
-- Looking for pathconf
-- Looking for pathconf - found
-- Looking for umask
-- Looking for umask - found
-- The C compiler identification is GNU 10.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done

Please see also: awslabs/aws-c-event-stream#15

[anuj812]

I am getting this issue as well for 1.9.19 as well. Where as it works perfectly for 1.8.186

[glaubitz]

Yep, it was introduced with version 1.9.0 which contains the commit bb96468.

[KaibaLopez]

Hi @glaubitz ,
Yea we got a few problems with adding the dependencies as submodules, still working on that.
Having said that I can make it work on my environment, so unless there is something specific going on in openSUSE (which I think shouldn't be the case) it could just be an environment configuration problem.
The most common cause of these errors is not having done the clone --recursive or if you already had version 1.8 git submodule update --init --recursive, remember to uninstall the previous crt version otherwise it will also fail.
If that doesn't work, can you walk me through every step you took to reproduce?

[glaubitz]

On May 20, 2021, at 7:24 PM, Gilberto Lopez Zayas ***@***.***> wrote: Hi @glaubitz , Yea we got a few problems with adding the dependencies as submodules, still working on that. Having said that I can make it work on my environment, so unless there is something specific going on in openSUSE (which I think shouldn't be the case) it could just be an environment configuration problem.

Well, I have been in touch with CMake upstream regarding the CMake issues with the various AWS C++ packages and they explicitly told me that the paths used and the way packages are included is wrong. I’ll try to loop in someone from CMake upstream in this discussion to help resolve this issue. FWIW, distributions don’t use git submodules but we rather split every dependency into a separate package to ease security maintenance.

[ncoder-1]

I'm having the same issue. I'm using the tar.gz release (and not a git clone).

For 1.9.25:

cd /tmp
curl https://codeload.github.com/aws/aws-sdk-cpp/tar.gz/refs/tags/1.9.25 -o aws-sdk-cpp-1.9.25.tar.gz
tar xvf aws-sdk-cpp-1.9.25.tar.gz
cd aws-sdk-cpp-1.9.25
mkdir build
cd build
cmake .. -G Ninja

Then I get the same error as above. This used to work with 1.8 releases. I'm also pulling specifc tar.gz releases in my own CMakeLists.txt files, so this is quite surprising that it stopped working.

[KaibaLopez]

@ncoder-1 ,
the tar.gz release is going to have the problem of not having the submodules included, for the moment this way of installing is not supported.
@glaubitz when you say:

distributions don’t use git submodules but we rather split every dependency into a separate package to ease security maintenance.

Does this mean that you just downloaded the project and built? (either via git or zip file), because then yea the same answer is going to apply to you and the only workaround for now is to keep using 1.8 since 1.9 requires the submodules to run.

[glaubitz]

Does this mean that you just downloaded the project and built?

Yes, downloaded from the tarball. Most distributions will use the tarball for creating a package, not check out the sources from git.

Distributions usually create individual packages for every upstream project which includes shared libraries. Having shared libraries in separate packages and not bundling them is an important security aspect for a distribution.

See: https://fedoraproject.org/wiki/Bundled_Libraries

[ncoder-1]

@KaibaLopez ,
So what is the use of the tar.gz releases if they are unsupported and not expected to work? For the reasons mentioned by @glaubitz , a lot of organizations require the use of tar.gz/zip releases on systems that build offline or for security reasons.

[KaibaLopez]

To clarify, I am not saying this is correct, I am trying to verify that was the cause and give a recommendation so this doesn't block you.
But yea, we are working to properly support this.

[Bu11etmagnet]

Building from .tar.gz is just as unsupported as building from git, i.e. it doesn't work.

[KaibaLopez]

Hi, as an update on this, we included a script that can be run to get the dependencies and build the sdk from the .zip file. See here

[ncoder-1]

HI,

I appreciate the gesture, but that won't allow for offline building if you are using curl to fetch content. It also becomes a nightmare in version management and will prevent reproducible builds which are more issues down the line...

[glaubitz]

I agree with @ncoder-1 . It's just a workaround and not addressing the actual problem which is the incorrect use of CMake module paths. FWIW, it can be helpful to ask CMake upstream (IRC, mailing list etc), they are very helpful when it comes to these problems.

[ncoder-1]

Any update or progress on this issue? I see tar.gz/zip "releases" being pushed out daily, which is surprising as you can't actually build them without a side script. I appreciate the time put into getting bug fixes, optimizations and improvements out, but I would hope that making the sdk releases buildable again would be a higher priority.

[anatol]

I see similar issue at Arch Linux:

-- Detecting CXX compile features - done
-- Found ZLIB: /usr/lib/libz.so (found version "1.2.11") 
--   Zlib library: /usr/lib/libz.so
-- Encryption: Openssl
-- Found OpenSSL: /usr/lib/libcrypto.so (found version "1.1.1l")  
--   Openssl include directory: /usr/include
--   Openssl library: /usr/lib/libssl.so;/usr/lib/libcrypto.so
-- Http client: Curl
-- Found CURL: /usr/lib/libcurl.so (found version "7.79.1")  
--   Curl include directory: /usr/include
--   Curl library: /usr/lib/libcurl.so
-- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC
-- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC - Success
CMake Error at CMakeLists.txt:202 (include):
  include could not find requested file:

    AwsFindPackage


-- Add s3-crt:s3 to C2J_SPECIAL_NAME_LIST
-- Looking for pathconf
-- Looking for pathconf - found
-- Looking for umask
-- Looking for umask - found
-- The C compiler identification is GNU 11.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
 -- Updating version info to 1.9.113
-- Custom memory management disabled
-- Performing Test CURL_HAS_H2
-- Performing Test CURL_HAS_H2 - Success
-- Performing Test CURL_HAS_TLS_PROXY
-- Performing Test CURL_HAS_TLS_PROXY - Success
CMake Error at aws-cpp-sdk-core/CMakeLists.txt:504 (aws_use_package):
  Unknown CMake command "aws_use_package".


-- Configuring incomplete, errors occurred!
See also "/build/aws-sdk-cpp/src/aws-sdk-cpp-1.9.113/build/CMakeFiles/CMakeOutput.log".
==> ERROR: A failure occurred in build().
    Aborting...

What is the way to fix/workaround this problem?

[r-burns]

@anatol you need to add the aws-c-common package's cmake files to CMAKE_MODULE_PATH. It looks like on arch linux these are sent to /usr/lib/cmake so -DCMAKE_MODULE_PATH=/usr/lib/cmake might do the trick.

Reading between the lines, if you're trying to bump arch's aws-sdk-cpp package to 1.9.x, feel free to take a look at my nixpkgs bump to see what other dependencies are required

[glaubitz]

@r-burns The problem is that the package ignores the common standard paths for cmake which is why it's so difficult to get it to build as a package. I have shown the CMakeLists.txt of one of the other AWS C++ packages to the cmake upstream developers and they pointed me the fact that the paths were wrong and the way stuff is included.

If properly done, it should work on just any distribution.

[r-burns]

Hey you're preaching to the choir here, I'm just throwing out some workarounds until this issue is resolved.

[anatol]

@r-burns thanks a lot for the reference to nixos package changes! It helped.

[glaubitz]

OK, I actually found the time for some digging now. I completely forgot that I was already patching CMAKE_MODULE_PATH myself. What I need to do is add a find_package() line for aws-crt-cpp. Should be trivial.

[glaubitz]

I have finally packaged all the dependencies in openSUSE now:

https://build.opensuse.org/project/show/devel:libraries:c_c++

However, I'm still not sure how to proceed with all the custom commands that the CMakeFiles.txt defines such as add_sdks().

@r-burns I tried your patch on openSUSE but it doesn't solve the issue for me.

[r-burns]

Wdym, what's the issue? Are you saying that the AwsFindPackage include still fails?

Not sure what you mean about add_sdks, afaict that function is only used at aws-sdk-cpp configure time so it doesn't need to be packaged.

[glaubitz]

Wdym, what's the issue? Are you saying that the AwsFindPackage include still fails?

Yes. The NixOS patch doesn't seem to be sufficient for current SDK versions.

Not sure what you mean about add_sdks, afaict that function is only used at aws-sdk-cpp configure time so it doesn't need to be packaged.

Well, it's the configure that fails ;).

[jmklix]

We have been working on improving the build process and have made some changes that might have fixed this already. You can track the progress here

[jmklix]

Can you trying updating the submodules to the latest version? You can do that with this:

git submodule update --init --recursive

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[r-burns]

Nope, still an issue. Aws-c-common adds modules to PREFIX/lib/cmake (which is nonstandard), which needs to be manually added to downstream packages' CMAKE_MODULE_PATH to be detected. A more proper mechanism would be to expose AwsCRuntime etc. through e.g. find_package(aws-c-common).

[hansalkothari]

This solution worked for me.

To check the submodules in the /aws-sdk-cpp run :
git submodule

To check their status, if they are installed:
git submodule status

You can manually the contents of the /home/glaubitz/upstream/aws-sdk-cpp/crt/aws-crt-cpp folder, whether it has CMakeLists.txt or not.

If there is nothing in the aws-crt-cpp, then do
cd ..
rm -rf aws-crt-cpp
git clone --recurse-submodules https://github.com/awslabs/aws-crt-cpp.git

The essence of the solution is, if a submodule is empty. Do a manual git clone

[glaubitz]

The essence of the solution is, if a submodule is empty. Do a manual git clone

Sorry, but this completely misses the point. First of all, a lot of packages of the AWS C++ SDK did not use the standard paths for placing CMake files as defined by upstream. Luckily, most packages have been fixed in the mean time.

And, secondly, I'm building distribution packages, so I cannot use git submodules.

Anyway, this works fine now and I guess this can be closed now.

[sbiscigl]

yes this was fixed with awslabs/aws-c-common#1178, will resolve, give a shout if you have any questions about it

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.