Skip to content

Commit f27e3a2

Browse files
committed
Integrate vcpkg to manage dependencies for all platforms (apache#371)
### Motivation Currently we manage dependencies by ourselves: - When running unit tests in CI, dependencies are installed from the Debian package management tool `apt` - When verifying macOS build, dependencies are installed from the Homebrew package manager - When verifying Windows build, dependencies are installed via vcpkg but no versions are specified - When building pre-built binaries to release, dependencies are installed by scripts under `pkg/` directories. These scripts download the source code according to `dependencies.yaml` and build It makes the pre-built binaries never tested except for the simple manual tests when voting for a new release. As a result, regression like apache#363 could happen. This patch aims at integrating vcpkg to manage dependencies for all platforms, not only for Windows. ### Modifications Integrate the CMakeLists.txt with vcpkg by: 1. Introduce vcpkg as a submodule of this project 2. Update `vcpkg.json` to specify dependency versions. 3. Set `CMAKE_TOOLCHAIN_FILE` with the `vcpkg.cmake` in the submodule. Then, we can simply use `find_package` to find all dependencies and depend on them via a target like `CURL::libcurl` to have all include directores and link libraries. Update the `unit-tests` workflow to verify building the binaries via vcpkg to test. The README is also updated to show how much simpler to build with vcpkg now. ### TODO There are still some tasks that cannot be done by vcpkg: 1. The static library (e.g. `libpulsarwithdeps.a`) that bundles all 3rd-party dependencies. 2. The pre-built binaries are still built by scripts under `./pkg` directory. Specifically, vcpkg does not work with GCC 4.8 so on CentOS 7 we still need to build dependencies manually. So the previous CMakeLists.txt are retained and will be used if `INTEGRATE_VCPKG` is OFF (by default). They will be removed until the task above can be done by vcpkg in future. We also need to update `dependencies.yaml` to be consistent with `vcpkg.json` if all tasks above cannot be replaced by vcpkg. (cherry picked from commit 7baa312)
1 parent ecb29f3 commit f27e3a2

16 files changed

+1020
-620
lines changed

.github/workflows/ci-pr-validation.yaml

+23-23
Original file line numberDiff line numberDiff line change
@@ -65,40 +65,40 @@ jobs:
6565
steps:
6666
- name: checkout
6767
uses: actions/checkout@v3
68+
with:
69+
fetch-depth: 0
70+
submodules: recursive
6871

69-
- name: Install deps
70-
run: |
71-
sudo apt-get update -y && \
72-
sudo apt-get install -y \
73-
libcurl4-openssl-dev \
74-
protobuf-compiler \
75-
libprotobuf-dev \
76-
libboost-dev \
77-
libboost-program-options-dev \
78-
libzstd-dev \
79-
libsnappy-dev \
80-
libgmock-dev \
81-
libgtest-dev
82-
83-
- name: Install gtest-parallel
72+
- name: Build core libraries
8473
run: |
85-
sudo curl -o /gtest-parallel https://raw.githubusercontent.com/google/gtest-parallel/master/gtest_parallel.py
86-
87-
- name: CMake
88-
run: cmake . -DCMAKE_BUILD_TYPE=Debug -DBUILD_PERF_TOOLS=ON
74+
cmake . -DINTEGRATE_VCPKG=ON -DBUILD_TESTS=OFF
75+
cmake --build . -j8
8976
9077
- name: Check formatting
91-
run: make check-format
78+
run: |
79+
./vcpkg/vcpkg format-manifest vcpkg.json
80+
if [[ $(git diff | wc -l) -gt 0 ]]; then
81+
echo "Please run `./vcpkg/vcpkg format-manifest vcpkg.json` to reformat vcpkg.json"
82+
fi
83+
make check-format
9284
93-
- name: Build
85+
- name: Build tests
9486
run: |
95-
# Build the libraries first to avoid possible link failures
96-
cmake --build . -j8 --target pulsarShared pulsarStatic
87+
cmake . -DINTEGRATE_VCPKG=ON -DBUILD_TESTS=ON
9788
cmake --build . -j8
9889
90+
- name: Install gtest-parallel
91+
run: |
92+
sudo curl -o /gtest-parallel https://raw.githubusercontent.com/google/gtest-parallel/master/gtest_parallel.py
93+
9994
- name: Run unit tests
10095
run: RETRY_FAILED=3 ./run-unit-tests.sh
10196

97+
- name: Build perf tools
98+
run: |
99+
cmake . -DINTEGRATE_VCPKG=ON -DBUILD_TESTS=ON -DBUILD_PERF_TOOLS=ON
100+
cmake --build . -j8
101+
102102
cpp20-build:
103103
name: Build with the C++20 standard
104104
runs-on: ubuntu-22.04

.gitmodules

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "vcpkg"]
2+
path = vcpkg
3+
url = https://github.com/microsoft/vcpkg.git

0 commit comments

Comments
 (0)