Repository navigation
docs: clarify executor container sizing and memory failures - #7332
Conversation
…at native-heap trimming
…contributor guide
andygrove
left a comment
There was a problem hiding this comment.
The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.
I did not find any blocking concerns. This is a documentation-only change, so there is no behavior difference against the latest release branch.
I checked the new container sizing sum against ResourceProfile.getResourcesForClusterManager in Spark 4.1.3. The total there is executor memory plus overhead plus off-heap plus the PySpark memory, and the PySpark term is only added when isPythonApp is true. That matches the new text in tuning/memory.md. The -XX:OnOutOfMemoryError='kill %p' claim matches YarnSparkHadoopUtil.addOutOfMemoryErrorArgument, including the non-Windows branch and the "unless already set" condition. The two example messages in the managed memory pool row, Additional allocation failed for and Not enough memory to continue external sort, both exist in DataFusion's pool and sort code. The ResourcesExhausted classification test in native/jni-bridge/src/errors.rs confirms they surface as CometNativeException. The new anchors and the relative link to memory_management.md resolve, and the wording changes in the contributor guide agree with the user guide.
andygrove
left a comment
There was a problem hiding this comment.
LGTM. Thanks @Abhishek-Mohan
Which issue does this PR close?
Closes #6032.
Rationale for this change
The memory tuning page explains how to size the executor memory overhead, but it never showed what the executor container is made of. It also gave no way to tell a container kill from heap exhaustion or a refused memory reservation. A reader who sees an executor die can easily raise the setting that has nothing to do with the failure.
The issue's last comment lists four things still to do. #6343 has since rewritten the introduction, so this PR covers the other three: the container sizing sum, a table of what each budget looks like when it is exceeded, and a link to the contributor guide's memory management page.
What changes are included in this PR?
docs/source/user-guide/latest/tuning/memory.md:spark.memory.offHeap.size, plusspark.executor.pyspark.memorywhen it applies. It says when the sum holds: Kubernetes and YARN, off-heap enabled, and the default resource profile.spark.memory.offHeap.sizegrows both the container and the shared reservation budget, while raisingspark.executor.memoryOverheadgrows only the container.-XX:OnOutOfMemoryError='kill %p'unless the Java options already set it, so a heap OOM there can exit with 143 rather than 52.docs/source/contributor-guide/memory_management.md:How are these changes tested?
Documentation only.
prettier --check(3.9.9) andgit diff --checkpass.review-comet-docsskill in PR mode. Its base and head Sphinx builds, in nitpicky mode with the expression templates copied in, both have 32 warnings, none new. Its config-key, wording, version and link checks are clean.ResourceProfile.getResourcesForClusterManager,Executor.isFatalError,SparkUncaughtExceptionHandler,YarnSparkHadoopUtil.addOutOfMemoryErrorArgument), Comet'sCometExecIteratorandnative/jni-bridge/src/errors.rs, and DataFusion 55.2.0's memory pool error messages.branch-1.2, there is no runtime behavior change.Council review
Each delegate ran
review-comet-pr,review-comet-docsandreview-comet-memory-pron the PR. Round 1 found two things: the contributor guide left out "non-Windows" for the YARN OOM handler, and this description listed only one changed file. Round 2 found that this description misstated which items of #6032 were already done. All three are fixed.Rounds: 3
AI assistance: Claude Opus 5.5 and GPT-6 Astra in OpenCode and T3 Code wrote and checked the change. The models above reviewed it.