Skip to content

docs: simplify the memory tuning page - #6343

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:docs-simplify-memory-tuning
Sep 28, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
andygrove:docs-simplify-memory-tuning

Conversation

@andygrove

@andygrove andygrove commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue; this is documentation only.

Rationale for this change

The memory tuning page had two answers to how much memory Comet needs. The older one, "Determining How Much Memory to Allocate", was added in March 2025, before Comet 0.8.0 was released. Its advice rests on TPC-H timings from a single executor at that time, and many memory bugs have been fixed since. The newer one, "Sizing the Overhead from the Memory Usage Log", uses the memory usage log added in 1.1.0 to measure what Comet actually uses on the user's own workload.

The page also explained untracked memory twice: once under Configuring Comet Memory, and again at the start of Configuring Executor Memory Overhead. And it carried a note about a fair_unified bug in Comet 0.15.0 through 1.0.0, which is advice for upgrading rather than for tuning.

What changes are included in this PR?

  • Removes the "Determining How Much Memory to Allocate" section and the intro paragraph that pointed to it. The intro now points to the memory usage log section instead.
  • Keeps the old section's one general point as a short paragraph under Configuring Comet Memory: a larger pool lets operators that can spill keep more in memory, and an operator that cannot spill fails the task when it cannot reserve the memory it needs. Nothing else on the page said why the pool's size matters.
  • Condenses the three opening paragraphs of Configuring Executor Memory Overhead into one, since they repeated the untracked-memory list from the section above. The paragraph keeps the two points specific to the overhead: the container already has room for spark.memory.offHeap.size, and the JVM's own non-heap memory draws on the same overhead.
  • Moves the note about the fair_unified pool in Comet 0.15.0 through 1.0.0 (fair_unified memory pool caps the pool total at pool_size / num_consumers instead of each consumer's share #5961) to a new Memory Pool Limits subsection under Upgrading to Comet 1.1.0 in the upgrade guide. The moved text now names spark.comet.exec.memoryPool and says what 1.1.0 changed: each operator is limited to its own share.

The headings that other pages link to are unchanged. The upgrade guide links to four of them, and the contributor guide links to the memory usage log section. Nothing linked to the removed section.

How are these changes tested?

Documentation only, no code paths touched.

I built the site locally with Sphinx before and after the change, and the change adds no warnings. MyST warns about any link to a missing heading anchor, and in the built HTML all eight links into this page's sections, from the upgrade guide and the contributor guide, reach an existing anchor. No page linked to the removed section. npx prettier@latest --check passes on both pages.

Remove the "Determining How Much Memory to Allocate" section. Its TPC-H
timings date from March 2025, before Comet 0.8.0 and many memory fixes,
and the memory usage log section now covers measuring how much memory
Comet uses. Keep its one general point, that a larger pool means fewer
spills, as a sentence under Configuring Comet Memory.

Also drop the intro paragraph that pointed to the removed section, and
condense the start of Configuring Executor Memory Overhead, which
repeated the untracked-memory explanation from the section above it.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 28, 2026
The paragraph about the fair_unified pool capping a task's operators at
one operator's share in Comet 0.15.0 through 1.0.0 (apache#5961) is upgrade
advice rather than tuning advice. Move it to a new Memory Pool Limits
subsection of Upgrading to Comet 1.1.0, naming the pool setting and
saying what 1.1.0 changed.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

  • Prior state and problem: The memory guide mixed historical benchmark advice with workload-based sizing guidance and repeated explanations of untracked memory.
  • Design approach: Remove the historical benchmark section, consolidate overhead guidance, and move the release-specific fair_unified warning into the migration guide.
  • Correctness / compatibility analysis: The revised claims match Comet’s reservation checks and configuration defaults. Spark’s task limits and Kubernetes/YARN container accounting were checked against sources for versions 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0.
  • Key design decisions: Keep pool reservations distinct from physical memory usage and preserve the explanation of spilling versus task failure. The consolidation reduces duplication without adding abstractions.
  • Implementation sketch: Two documentation files change across two commits. Existing linked headings remain, and Memory Pool Limits is added under upgrading to Comet 1.1.0.
  • Behavioral changes worth calling out: Readers are directed to workload measurements instead of historical benchmark figures. Runtime behavior, defaults, and performance overhead are unchanged.
  • Suggested improvements: None meeting the P1/P2 threshold. No introduced P1/P2 issues found within this review.

Reviewed the entire diff from base e897f8ab45dec8fc27c79339c65eeaa7bd60e198 through head e00c8e78114128af0d178762b6f52345cfb91b99. The PR is not a draft. Routed skills: review-comet-pr and review-comet-memory-pr. The snapshot and live discussion contained no reviews, issue comments, inline comments, or review threads.

Exact-head CI: Seven checks passed, including Required Checks. Fifteen checks were skipped, including runtime builds, benchmarks, Spark/Iceberg suites, and site deployment. An earlier title check was cancelled before a successful replacement. No failed checks were reported.

Validation: git diff --check passed. A direct Sphinx HTML build succeeded, and all eight inbound links to memory-page headings plus twelve internal anchor links resolved. Neither edited page produced warnings. Validation limits: Mermaid rendering was disabled, generated/version documentation was not produced, and the build reported 60 warnings in unchanged pages for missing documents and unresolved references. The full publication build and JVM/native test suites were not run.

@andygrove
andygrove added this pull request to the merge queue Sep 28, 2026
Merged via the queue into apache:main with commit 22a0706 Sep 28, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants