Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Paimon C++ currently only builds and is validated on x86_64. aarch64/arm64 has become a mainstream server and desktop platform (Apple Silicon, AWS Graviton, Alibaba Cloud Yitian, Ampere, etc.), and users deploying Paimon readers/writers on these platforms cannot use the library today.
Concretely, the following gaps block first-class aarch64/arm64 support:
- No architecture detection in the build system.
cmake_modules/SetupCxxFlags.cmakeconsumesPAIMON_CPU_FLAG(withx86,ppc, andarmv8branches), but nothing in the repository ever sets it fromCMAKE_SYSTEM_PROCESSOR, so thearmv8tuning path (PAIMON_ARMV8_ARCH_FLAG, CRC feature detection) is effectively dead code and the x86 SIMD defines (PAIMON_HAVE_SSE4_2/AVX2/AVX512) are not guarded by an actual architecture check. - Third-party dependencies with x86_64-only artifacts. In
cmake_modules/ThirdpartyToolchain.cmake, the lumina dependency hardcodesartifacts/cpp/linux-x86_64/install-root/...paths, so the build cannot link it on aarch64. (JindoSDK already resolvesaarch64artifacts for both Linux and macOS, so it is not a blocker.) The tantivy-based global index is built from Rust via Corrosion and needs verification that the toolchain targets aarch64 correctly. - No aarch64 CI coverage. All workflows under
.github/workflows/run on x86_64 GitHub-hosted runners (ubuntu-22.04/ubuntu-24.04), so architecture-specific regressions are invisible. There is at least one known class of portability issues already noted in the codebase:charis unsigned by default on aarch64 (see the workaround comment incmake_modules/BuildUtils.cmake), which can silently change behavior of code assuming signedchar. - No arm64 release artifacts. The packaging/release tooling only produces x86_64 outputs, and the documentation does not state supported platforms.
Solution
Proposed incremental plan:
- Derive
PAIMON_CPU_FLAGfromCMAKE_SYSTEM_PROCESSOR(x86/armv8/ppc) in the CMake setup, so the existing armv8 branch inSetupCxxFlags.cmakebecomes reachable and x86 SIMD flags are only applied on x86. - Audit dependencies in
ThirdpartyToolchain.cmake: select aarch64 artifacts where available (as already done for JindoSDK), build from source or make the dependency optional where no aarch64 artifact exists (e.g. lumina), and verify the Corrosion/Rust tantivy build on aarch64. - Add an aarch64 CI job using GitHub's arm64 hosted runners (
ubuntu-24.04-arm) that runs the standard build and unit test suite, and fix any test failures uncovered (char signedness, alignment, intrinsics). - Extend the release/packaging scripts to produce linux-aarch64 (and macos-arm64) artifacts, and document the supported platform matrix in
docs/source/building.rst.
Anything else?
No response
Are you willing to submit a PR?
- I'm willing to submit a PR!
Reacted by lxy
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity