Skip to content

docs: clarify executor container sizing and memory failures - #7332

Merged
andygrove merged 10 commits into
apache:mainfrom
Abhishek-Mohan:docs/6032-memory-tuning-gaps
Nov 14, 2026
Merged

andygrove merged 10 commits into
apache:mainfrom
Abhishek-Mohan:docs/6032-memory-tuning-gaps

Conversation

@Abhishek-Mohan

@Abhishek-Mohan Abhishek-Mohan commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6032.

Rationale for this change

The memory tuning page explains how to size the executor memory overhead, but it never showed what the executor container is made of. It also gave no way to tell a container kill from heap exhaustion or a refused memory reservation. A reader who sees an executor die can easily raise the setting that has nothing to do with the failure.

The issue's last comment lists four things still to do. #6343 has since rewritten the introduction, so this PR covers the other three: the container sizing sum, a table of what each budget looks like when it is exceeded, and a link to the contributor guide's memory management page.

What changes are included in this PR?

docs/source/user-guide/latest/tuning/memory.md:

  • Adds Spark's container sizing sum: executor memory, plus overhead, plus spark.memory.offHeap.size, plus spark.executor.pyspark.memory when it applies. It says when the sum holds: Kubernetes and YARN, off-heap enabled, and the default resource profile.
  • Explains that raising spark.memory.offHeap.size grows both the container and the shared reservation budget, while raising spark.executor.memoryOverhead grows only the container.
  • Replaces the advice to raise overhead whenever an executor exits with code 137, which can also be a kill for another reason.
  • Adds a "Diagnosing Memory Failures" table for the three budgets: container memory, JVM heap, and the managed memory pool. Each row gives the signals and what to change. The container row points to the existing section on returning freed native memory, for executors killed while the memory usage log stays well within the limits.
  • Notes that on non-Windows YARN executors Spark adds -XX:OnOutOfMemoryError='kill %p' unless the Java options already set it, so a heap OOM there can exit with 143 rather than 52.
  • Links to the contributor guide's Memory Management page.

docs/source/contributor-guide/memory_management.md:

  • The overview and the OOM triage checklist said the exit code alone identifies the budget. They now say to confirm it from the cluster manager's diagnostics and the executor log, matching the user guide.

How are these changes tested?

Documentation only.

  • prettier --check (3.9.9) and git diff --check pass.
  • I ran the review-comet-docs skill in PR mode. Its base and head Sphinx builds, in nitpicky mode with the expression templates copied in, both have 32 warnings, none new. Its config-key, wording, version and link checks are clean.
  • I checked the claims against Spark 4.1.3 (ResourceProfile.getResourcesForClusterManager, Executor.isFatalError, SparkUncaughtExceptionHandler, YarnSparkHadoopUtil.addOutOfMemoryErrorArgument), Comet's CometExecIterator and native/jni-bridge/src/errors.rs, and DataFusion 55.2.0's memory pool error messages.
  • Compared against branch-1.2, there is no runtime behavior change.

Council review

Each delegate ran review-comet-pr, review-comet-docs and review-comet-memory-pr on the PR. Round 1 found two things: the contributor guide left out "non-Windows" for the YARN OOM handler, and this description listed only one changed file. Round 2 found that this description misstated which items of #6032 were already done. All three are fixed.

Rounds: 3

Delegate Round 1 Round 2 Round 3
Claude Opus 5.5 9/10 9/10 10/10
GPT-6.1 Sol 9/10 10/10 10/10
Gemini 4 Argon 10/10 10/10 10/10

AI assistance: Claude Opus 5.5 and GPT-6 Astra in OpenCode and T3 Code wrote and checked the change. The models above reviewed it.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Oct 30, 2026
@Abhishek-Mohan
Abhishek-Mohan marked this pull request as ready for review November 6, 2026 19:02

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR backlog is huge and the maintainers don't have the bandwidth to review them all, so here is an automated review. Feel free to ignore it.

I did not find any blocking concerns. This is a documentation-only change, so there is no behavior difference against the latest release branch.

I checked the new container sizing sum against ResourceProfile.getResourcesForClusterManager in Spark 4.1.3. The total there is executor memory plus overhead plus off-heap plus the PySpark memory, and the PySpark term is only added when isPythonApp is true. That matches the new text in tuning/memory.md. The -XX:OnOutOfMemoryError='kill %p' claim matches YarnSparkHadoopUtil.addOutOfMemoryErrorArgument, including the non-Windows branch and the "unless already set" condition. The two example messages in the managed memory pool row, Additional allocation failed for and Not enough memory to continue external sort, both exist in DataFusion's pool and sort code. The ResourcesExhausted classification test in native/jni-bridge/src/errors.rs confirms they surface as CometNativeException. The new anchors and the relative link to memory_management.md resolve, and the wording changes in the contributor guide agree with the user guide.

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @Abhishek-Mohan

@andygrove
andygrove added this pull request to the merge queue Nov 14, 2026
Merged via the queue into apache:main with commit f043d64 Nov 14, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tuning guide does not explain that Comet's memory comes out of the executor container

2 participants