Skip to content

Comet executor memory overriding to absurd numbers (unified mode) #1460

Closed

Description

@LukMRVC

Describe the bug

When initialization Spark session in unified mode, Comet overrides spark.executor.memoryOverhead to over 100GB memory requirements.

val cometMemOverhead =
if (!CometSparkSessionExtensions.cometUnifiedMemoryManagerEnabled(sc.getConf)) {
CometSparkSessionExtensions.getCometMemoryOverheadInMiB(sc.getConf)
} else {
// comet shuffle unified memory manager is disabled, so we need to add overhead memory
CometSparkSessionExtensions.getCometShuffleMemorySize(sc.getConf)
}
sc.conf.set(EXECUTOR_MEMORY_OVERHEAD.key, s"${execMemOverhead + cometMemOverhead}M")

First the if clause seems to be reversed. Secondly, CometSparkSessionExtensions.getCometShuffleMemorySize returns memory in bytes, which is added to memory in MB. This leads overriding memory allocation to absurd sizes.

Steps to reproduce

Create spark session and see spark.executor.memoryOverhead.

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
    .appName("sample") 
    .config("spark.executor.memory", "2G")
    .config("spark.executor.memoryOverhead", "5G")
    .config("spark.driver.memory", "2G") 
    .config("spark.driver.memoryOverhead", "2G")
    
    .config("spark.comet.exec.enabled", "true")
    .config("spark.shuffle.manager", "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
    .config("spark.comet.shuffle.enabled", "true")
    .config("spark.comet.explainFallback.enabled", "true")
    .config("spark.memory.offHeap.enabled", "true")
    .config("spark.memory.offHeap.size", "3G")
    .config("spark.comet.memory.overhead.factor", "0.2")    
    .config("spark.comet.convert.json.enabled", "true")
    
    .config("spark.plugins", "org.apache.spark.CometPlugin")
    .config("spark.comet.enabled", "true")
    .getOrCreate()

Expected behavior

Overriding executor memory to reasonable amounts set by spark.comet.memoryOverhead or computed in MB units

Additional context

No response

Activity

  1. self-assigned this
    on Feb 28, 2025
  2. added this to the 0.7.0 milestone on Feb 28, 2025
  3. andygrove commented on Feb 28, 2025

    @andygrove
    Member

    Thanks for the bug report @LukMRVC. I will investigate this today.

  4. LukMRVC commented on Feb 28, 2025

    @LukMRVC
    ContributorAuthor

    If you don't mind, I could try to fix, I'd like to contribute to the project, seems like a nice goof first issue.

  5. andygrove commented on Feb 28, 2025

    @andygrove
    Member

    If you don't mind, I could try to fix, I'd like to contribute to the project, seems like a nice goof first issue.

    That would be awesome. I'll be happy to review.

  6. LukMRVC commented on Mar 2, 2025

    @LukMRVC
    ContributorAuthor

    Added a PR. There were some semantic issues in if clauses. Conf value
    COMET_COLUMNAR_SHUFFLE_UNIFIED_MEMORY_ALLOCATOR_IN_TEST would cause a tautology in an if condition checking for running in unified memory mode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions