Repository navigation
Description
What is the problem the feature request solves?
Rust 1.100 stabilized the core of the Allocator API: the Allocator trait, Global/System, and allocator-aware Vec and Box (new_in, with_capacity_in, allocator(), raw-parts variants). See the release announcement.
Comet charges memory-pool reservations for buffers it owns by estimating sizes manually, and the tests observe allocations by installing a crate-wide #[global_allocator]. A custom Allocator could make that accounting exact and let buffer growth fail gracefully under memory pressure. This issue is to research whether that is worth doing.
Describe the potential solution
Ideas to evaluate, in priority order:
- Pool-charging allocator for Comet-owned byte buffers. An
Allocatorthat grows and shrinks aMemoryReservation(or calls theMemoryPooldirectly) on allocate and deallocate. Use it for the shuffle path's ownVec<u8>/Box<[u8]>buffers, e.g.shuffle/src/ipc.rs, the scratch buffers inwriters/buf_batch_writer.rs, and theCursor<Vec<u8>>inwriters/rss/rss_partition_writer.rs.- Accounting would be exact instead of estimated (see the note near
shuffle_writer.rs:1082). Vec::try_reserveworks with any allocator, so pool exhaustion could become anErrthat triggers a spill instead of over-allocating.- Open questions:
Allocatormethods take&selfbutMemoryReservation::try_growneeds&mut self. Infallible calls (push,extend) panic onAllocError, so hot paths would needtry_reserve.
- Accounting would be exact instead of estimated (see the note near
- Tests that check memory estimates against real allocations. Much of Comet's pool accounting reserves an estimated size before allocating. Tests could route the buffers through a counting allocator and assert that the estimate is within a tolerance of the bytes actually allocated, including peak usage during growth and realloc. That would catch estimates that drift as the code changes, and it could start before (1) is adopted in production code.
- Scoped counting allocators in tests. Replace the crate-global
ObservedAllocatorinshuffle/src/writers/rss/mod.rsand theSERIALlock incore/src/alloc_accounting.rswith per-test allocators. This only works once the code under test is generic over the allocator. - Keep
AccountingAllocator(core/src/alloc_accounting.rs) as aGlobalAlloc. It provides the process-wide view, which the new API does not replace.
Additional context
- Arrow buffers are out of scope. arrow-rs
MutableBufferallocates directly and does not take anAllocator, so most of Comet's memory is not reachable through this API. Allocator support in arrow-rs would be a separate upstream discussion. StringandHashMapare not allocator-aware in std yet.- MSRV:
native/Cargo.tomlsetsrust-version = "1.95.0". Using the std API means raising it to 1.100, in line with DataFusion's MSRV policy. - Prototype on stable now: the
allocator-api2crate provides the same API on current stable Rust, so the research can start before any MSRV bump, then switch tostd::alloc::Allocatorlater.
Suggested first step: prototype (1) on the shuffle write path with allocator-api2. Measure how closely reservations track actual buffer bytes, and whether try_reserve-driven spilling behaves correctly under pressure.
Activity