Repository navigation
Closed
Description
Describe the bug
When initialization Spark session in unified mode, Comet overrides spark.executor.memoryOverhead to over 100GB memory requirements.
datafusion-comet/spark/src/main/scala/org/apache/spark/Plugins.scala
Lines 65 to 72 in 928e1a2
| val cometMemOverhead = | |
| if (!CometSparkSessionExtensions.cometUnifiedMemoryManagerEnabled(sc.getConf)) { | |
| CometSparkSessionExtensions.getCometMemoryOverheadInMiB(sc.getConf) | |
| } else { | |
| // comet shuffle unified memory manager is disabled, so we need to add overhead memory | |
| CometSparkSessionExtensions.getCometShuffleMemorySize(sc.getConf) | |
| } | |
| sc.conf.set(EXECUTOR_MEMORY_OVERHEAD.key, s"${execMemOverhead + cometMemOverhead}M") |
First the if clause seems to be reversed. Secondly, CometSparkSessionExtensions.getCometShuffleMemorySize returns memory in bytes, which is added to memory in MB. This leads overriding memory allocation to absurd sizes.
Steps to reproduce
Create spark session and see spark.executor.memoryOverhead.
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("sample")
.config("spark.executor.memory", "2G")
.config("spark.executor.memoryOverhead", "5G")
.config("spark.driver.memory", "2G")
.config("spark.driver.memoryOverhead", "2G")
.config("spark.comet.exec.enabled", "true")
.config("spark.shuffle.manager", "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager")
.config("spark.comet.shuffle.enabled", "true")
.config("spark.comet.explainFallback.enabled", "true")
.config("spark.memory.offHeap.enabled", "true")
.config("spark.memory.offHeap.size", "3G")
.config("spark.comet.memory.overhead.factor", "0.2")
.config("spark.comet.convert.json.enabled", "true")
.config("spark.plugins", "org.apache.spark.CometPlugin")
.config("spark.comet.enabled", "true")
.getOrCreate()Expected behavior
Overriding executor memory to reasonable amounts set by spark.comet.memoryOverhead or computed in MB units
Additional context
No response
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
andygrove commentedon Feb 28, 2025
Thanks for the bug report @LukMRVC. I will investigate this today.
LukMRVC commentedon Feb 28, 2025
If you don't mind, I could try to fix, I'd like to contribute to the project, seems like a nice goof first issue.
andygrove commentedon Feb 28, 2025
That would be awesome. I'll be happy to review.
LukMRVC commentedon Mar 2, 2025
Added a PR. There were some semantic issues in if clauses. Conf value
COMET_COLUMNAR_SHUFFLE_UNIFIED_MEMORY_ALLOCATOR_IN_TESTwould cause a tautology in an if condition checking for running in unified memory mode.