Skip to content

MemoryConsumer::try_grow Underflow  #4328

Closed

Description

@tustvold

Describe the bug

MemoryConsumer::try_grow calls through to MemoryManager::can_grow_directly which determines the remaining space by performing self.pool_size - trk_total.

This can potentially underflow if the tracked memory exceeds the pool size, something for which there is no protection.

To Reproduce

#[test]
fn test_underflow() {
    let config = MemoryManagerConfig::try_new_limit(100, 0.5).unwrap();
    let manager = MemoryManager::new(config);
    manager.grow_tracker_usage(100);

    manager.register_requester(&MemoryConsumerId::new(1));
    assert!(!manager.can_grow_directly(20, 0));
}

Expected behavior

Additional context

Activity

  1. askoa commented on Nov 23, 2022

    @askoa
    Contributor

    I can pick this up. I am planning to return zero if trk_total is more than self.pool_size in below function. Let me know what you think.

    https://github.com/apache/arrow-datafusion/blob/df8aa7a2e2a6f54acfbfed336b84144256fb7ff8/datafusion/core/src/execution/memory_manager/mod.rs#L331-L335

  2. alamb commented on Dec 19, 2022

    @alamb
    Contributor

    I think this is now complete with #4351 and #4522

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions