Repository navigation
Conversation
build-release-comet.sh now builds the native library twice per architecture, with the system allocator and with jemalloc, and packages every jar twice. The jar with the system allocator stays the artifact. The jemalloc jar is installed next to it with the jemalloc classifier, for example comet-spark-spark4.1_2.13-1.2.0-jemalloc.jar, so publish-to-maven.sh uploads it without changes. The jemalloc jars are packaged after a clean build rather than installed, because installing would replace the jars that use the system allocator. The script checks that every jar holds the native libraries built for its variant. The arm64 jemalloc library is built for 64 KiB pages so that it loads on every arm64 kernel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #7344.
Rationale for this change
#7356 added a
jemallocMaven profile, but the published jars still use the system allocator, which is glibc malloc on Linux. Users who want to try jemalloc in an executor, and anyone gathering the data that #7344 asks for, have to build Comet from source. Publishing a jemalloc jar with each release lets them swap the jar and compare. The jar with the system allocator stays the artifact, so nothing changes for existing users.What changes are included in this PR?
dev/release/comet-rm/build-comet-native-libs.shbuildslibcomet.sotwice in the same container, with the system allocator and with thejemallocCargo feature, and fails if the two builds are identical. jemalloc is built for 4 KiB memory pages on both architectures, set explicitly withJEMALLOC_SYS_WITH_LG_PAGE=12so that the library does not depend on the machine that built it. jemalloc fails every allocation on a kernel with larger pages, and some arm64 kernels use 64 KiB pages, for example RHEL 8 and its rebuilds. The release does not build for them, because a build for 64 KiB pages can raise memory use on the kernels with smaller pages, which are most of them. The user guide and the FAQ tell users on 64 KiB-page kernels to use the jar with the system allocator or to build from source withJEMALLOC_SYS_WITH_LG_PAGE=16.dev/release/build-release-comet.shcopies both sets of libraries out of the containers and builds every Spark and Scala combination twice. The jars with the system allocator are installed as before. The script then runsclean, packages the jemalloc libraries with-P jemalloc, and copies each jar into the staging repository with thejemallocclassifier, for examplecomet-spark-spark4.1_2.13-1.2.0-jemalloc.jar. These builds runpackagerather thaninstall, because installing would replace the jars with the system allocator. The jemalloc jar has no pom or sources jar of its own.publish-to-maven.shis unchanged: it already uploads and signs every.jarand.pomunder the staging repository, which includes the classifier jar.release_process.md, the script usage text,source.mdand the FAQ describe the two jars and the limitation on 64 KiB-page kernels.publish_snapshot.ymlis not changed, so snapshots still publish only the jar with the system allocator.How are these changes tested?
The Docker half of the release script takes hours and has not been run. The shell scripts pass
bash -n. For the Maven half, I builtspark-4.1with stand-in libraries on a throwaway checkout: a defaultpackage, thenclean, then a-P jemallocpackage. The jar name matches what the script copies,cleanremoves the stale libraries, and the jemalloc jar holds the jemalloc libraries. I also rancheck_native_libsagainst that jar. It passes for the matching variant and fails for the other one. The first release candidate built with these scripts is the end-to-end test.