Skip to content

Research using the Rust Allocator API (stabilized in 1.100) for memory-pool accounting #7613

Open

Description

@andygrove

What is the problem the feature request solves?

Rust 1.100 stabilized the core of the Allocator API: the Allocator trait, Global/System, and allocator-aware Vec and Box (new_in, with_capacity_in, allocator(), raw-parts variants). See the release announcement.

Comet charges memory-pool reservations for buffers it owns by estimating sizes manually, and the tests observe allocations by installing a crate-wide #[global_allocator]. A custom Allocator could make that accounting exact and let buffer growth fail gracefully under memory pressure. This issue is to research whether that is worth doing.

Describe the potential solution

Ideas to evaluate, in priority order:

  1. Pool-charging allocator for Comet-owned byte buffers. An Allocator that grows and shrinks a MemoryReservation (or calls the MemoryPool directly) on allocate and deallocate. Use it for the shuffle path's own Vec<u8>/Box<[u8]> buffers, e.g. shuffle/src/ipc.rs, the scratch buffers in writers/buf_batch_writer.rs, and the Cursor<Vec<u8>> in writers/rss/rss_partition_writer.rs.
    • Accounting would be exact instead of estimated (see the note near shuffle_writer.rs:1082).
    • Vec::try_reserve works with any allocator, so pool exhaustion could become an Err that triggers a spill instead of over-allocating.
    • Open questions: Allocator methods take &self but MemoryReservation::try_grow needs &mut self. Infallible calls (push, extend) panic on AllocError, so hot paths would need try_reserve.
  2. Tests that check memory estimates against real allocations. Much of Comet's pool accounting reserves an estimated size before allocating. Tests could route the buffers through a counting allocator and assert that the estimate is within a tolerance of the bytes actually allocated, including peak usage during growth and realloc. That would catch estimates that drift as the code changes, and it could start before (1) is adopted in production code.
  3. Scoped counting allocators in tests. Replace the crate-global ObservedAllocator in shuffle/src/writers/rss/mod.rs and the SERIAL lock in core/src/alloc_accounting.rs with per-test allocators. This only works once the code under test is generic over the allocator.
  4. Keep AccountingAllocator (core/src/alloc_accounting.rs) as a GlobalAlloc. It provides the process-wide view, which the new API does not replace.

Additional context

  • Arrow buffers are out of scope. arrow-rs MutableBuffer allocates directly and does not take an Allocator, so most of Comet's memory is not reachable through this API. Allocator support in arrow-rs would be a separate upstream discussion.
  • String and HashMap are not allocator-aware in std yet.
  • MSRV: native/Cargo.toml sets rust-version = "1.95.0". Using the std API means raising it to 1.100, in line with DataFusion's MSRV policy.
  • Prototype on stable now: the allocator-api2 crate provides the same API on current stable Rust, so the research can start before any MSRV bump, then switch to std::alloc::Allocator later.

Suggested first step: prototype (1) on the shuffle write path with allocator-api2. Measure how closely reservations track actual buffer bytes, and whether try_reserve-driven spilling behaves correctly under pressure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions