[ML] Add cross compilation support, Docker images and CI for aarch64 #1135
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Following on from elastic#1132, this change adds: 1. The ability to cross compile a Linux build for aarch64 on Linux on another hardware architecture 2. A Docker container for x86_64 that is set up for cross compiling for aarch64 3. A Docker container for aarch64 that can both build and run unit tests 4. Changes to CI scripts to utilise these Docker containers where appropriate
droberts195
commented
Apr 14, 2020
| @@ -192,13 +209,7 @@ case `uname` in | |||
| if [ $? -eq 0 ] ; then | |||
| echo "Set RPATH in $FILE" | |||
| else | |||
| # Set RPATH for 3rd party libraries that reference other libraries we ship | |||
| ldd $FILE | grep /usr/local/lib >/dev/null 2>&1 && patchelf --set-rpath '$ORIGIN/.' $FILE | |||
There was a problem hiding this comment.
I removed this bit because it hasn't been doing anything since we switched from storing the 3rd party libraries we build from /usr/local/lib to /usr/local/gcc75/lib. This came up in this PR because ldd only works on the native architecture. But the line is completely unnecessary now as Boost must have changed the way they link related libraries since we first added this line.
tveasey
approved these changes
Apr 15, 2020
| SYSROOT=/usr/local/sysroot-$CPP_CROSS_COMPILE-linux-gnu | ||
| BOOST_LOCATION=$SYSROOT/usr/local/gcc75/lib | ||
| BOOST_COMPILER=gcc | ||
| if [ "$CPP_CROSS_COMPILE" = aarch64 ] ; then |
There was a problem hiding this comment.
Is there any reason to set SYSROOT, etc if this if condition fails. It seems more natural to just add this as a new
elif [ "$CPP_CROSS_COMPILE" = aarch64 ] ; then and keep the failure case at the end? Then if we added another target we could just add a new elif.
There was a problem hiding this comment.
The reason I did it like this is that during the porting exercise I realised that a port to Linux on any other architecture could reuse a lot of the same code. Everything between lines 78 and 96 would be the same except the Boost abbreviation for the hardware architecture.
(This is also why I named the new make rules file linux_cross_compile_linux.mk. I started out with linux_cross_compile_aarch64.mk but realised before I opened the PR that almost the entire file would be the same for a cross compile to Linux on any other hardware architecture. Maybe in 5 years time we'll be cross compiling x86_64 from aarch64.)
Co-Authored-By: Tom Veasey <[email protected]>
|
retest |
droberts195
added a commit
to droberts195/ml-cpp
that referenced
this pull request
Apr 15, 2020
…ch64 Following on from elastic#1134, this change adds: 1. The ability to cross compile a Linux build for aarch64 on Linux on another hardware architecture 2. A Docker container for x86_64 that is set up for cross compiling for aarch64 3. A Docker container for aarch64 that can both build and run unit tests 4. Changes to CI scripts to utilise these Docker containers where appropriate Backport of elastic#1135
This was referenced Apr 15, 2020
droberts195
added a commit
to droberts195/elasticsearch
that referenced
this pull request
Apr 15, 2020
Following elastic/ml-cpp#1135 there are now Linux binaries for both x86_64 and aarch64. The code that finds the correct binaries to ship with each distribution was including both on every Linux distribution. This change alters that logic to consider the architecture as well as the operating system. Also, there is no need to disable ML on aarch64 now that we have the native binaries available. ML is still not supported on aarch64, but the processes at least run up and work at a superficial level.
droberts195
added a commit
to elastic/elasticsearch
that referenced
this pull request
Apr 15, 2020
Following elastic/ml-cpp#1135 there are now Linux binaries for both x86_64 and aarch64. The code that finds the correct binaries to ship with each distribution was including both on every Linux distribution. This change alters that logic to consider the architecture as well as the operating system. Also, there is no need to disable ML on aarch64 now that we have the native binaries available. ML is still not supported on aarch64, but the processes at least run up and work at a superficial level.
droberts195
added a commit
to droberts195/elasticsearch
that referenced
this pull request
Apr 16, 2020
Following elastic/ml-cpp#1135 there are now Linux binaries for both x86_64 and aarch64. The code that finds the correct binaries to ship with each distribution was including both on every Linux distribution. This change alters that logic to consider the architecture as well as the operating system. Also, there is no need to disable ML on aarch64 now that we have the native binaries available. ML is still not supported on aarch64, but the processes at least run up and work at a superficial level. Backport of elastic#55256
droberts195
added a commit
to elastic/elasticsearch
that referenced
this pull request
Apr 16, 2020
Following elastic/ml-cpp#1135 there are now Linux binaries for both x86_64 and aarch64. The code that finds the correct binaries to ship with each distribution was including both on every Linux distribution. This change alters that logic to consider the architecture as well as the operating system. Also, there is no need to disable ML on aarch64 now that we have the native binaries available. ML is still not supported on aarch64, but the processes at least run up and work at a superficial level. Backport of #55256
droberts195
added a commit
that referenced
this pull request
Apr 16, 2020
…ch64 (#1138) Following on from #1134, this change adds: 1. The ability to cross compile a Linux build for aarch64 on Linux on another hardware architecture 2. A Docker container for x86_64 that is set up for cross compiling for aarch64 3. A Docker container for aarch64 that can both build and run unit tests 4. Changes to CI scripts to utilise these Docker containers where appropriate Backport of #1135
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Following on from #1132, this change adds:
Linux on another hardware architecture
compiling for aarch64
unit tests
where appropriate