Repository navigation
docs: simplify the memory tuning page - #6343
Conversation
Remove the "Determining How Much Memory to Allocate" section. Its TPC-H timings date from March 2025, before Comet 0.8.0 and many memory fixes, and the memory usage log section now covers measuring how much memory Comet uses. Keep its one general point, that a larger pool means fewer spills, as a sentence under Configuring Comet Memory. Also drop the intro paragraph that pointed to the removed section, and condense the start of Configuring Executor Memory Overhead, which repeated the untracked-memory explanation from the section above it.
The paragraph about the fair_unified pool capping a task's operators at one operator's share in Comet 0.15.0 through 1.0.0 (apache#5961) is upgrade advice rather than tuning advice. Move it to a new Memory Pool Limits subsection of Upgrading to Comet 1.1.0, naming the pool setting and saying what 1.1.0 changed.
sunchao
left a comment
There was a problem hiding this comment.
Summary
- Prior state and problem: The memory guide mixed historical benchmark advice with workload-based sizing guidance and repeated explanations of untracked memory.
- Design approach: Remove the historical benchmark section, consolidate overhead guidance, and move the release-specific
fair_unifiedwarning into the migration guide. - Correctness / compatibility analysis: The revised claims match Comet’s reservation checks and configuration defaults. Spark’s task limits and Kubernetes/YARN container accounting were checked against sources for versions 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0.
- Key design decisions: Keep pool reservations distinct from physical memory usage and preserve the explanation of spilling versus task failure. The consolidation reduces duplication without adding abstractions.
- Implementation sketch: Two documentation files change across two commits. Existing linked headings remain, and
Memory Pool Limitsis added under upgrading to Comet 1.1.0. - Behavioral changes worth calling out: Readers are directed to workload measurements instead of historical benchmark figures. Runtime behavior, defaults, and performance overhead are unchanged.
- Suggested improvements: None meeting the P1/P2 threshold. No introduced P1/P2 issues found within this review.
Reviewed the entire diff from base e897f8ab45dec8fc27c79339c65eeaa7bd60e198 through head e00c8e78114128af0d178762b6f52345cfb91b99. The PR is not a draft. Routed skills: review-comet-pr and review-comet-memory-pr. The snapshot and live discussion contained no reviews, issue comments, inline comments, or review threads.
Exact-head CI: Seven checks passed, including Required Checks. Fifteen checks were skipped, including runtime builds, benchmarks, Spark/Iceberg suites, and site deployment. An earlier title check was cancelled before a successful replacement. No failed checks were reported.
Validation: git diff --check passed. A direct Sphinx HTML build succeeded, and all eight inbound links to memory-page headings plus twelve internal anchor links resolved. Neither edited page produced warnings. Validation limits: Mermaid rendering was disabled, generated/version documentation was not produced, and the build reported 60 warnings in unchanged pages for missing documents and unresolved references. The full publication build and JVM/native test suites were not run.
Which issue does this PR close?
No issue; this is documentation only.
Rationale for this change
The memory tuning page had two answers to how much memory Comet needs. The older one, "Determining How Much Memory to Allocate", was added in March 2025, before Comet 0.8.0 was released. Its advice rests on TPC-H timings from a single executor at that time, and many memory bugs have been fixed since. The newer one, "Sizing the Overhead from the Memory Usage Log", uses the memory usage log added in 1.1.0 to measure what Comet actually uses on the user's own workload.
The page also explained untracked memory twice: once under Configuring Comet Memory, and again at the start of Configuring Executor Memory Overhead. And it carried a note about a
fair_unifiedbug in Comet 0.15.0 through 1.0.0, which is advice for upgrading rather than for tuning.What changes are included in this PR?
spark.memory.offHeap.size, and the JVM's own non-heap memory draws on the same overhead.fair_unifiedpool in Comet 0.15.0 through 1.0.0 (fair_unified memory pool caps the pool total at pool_size / num_consumers instead of each consumer's share #5961) to a new Memory Pool Limits subsection under Upgrading to Comet 1.1.0 in the upgrade guide. The moved text now namesspark.comet.exec.memoryPooland says what 1.1.0 changed: each operator is limited to its own share.The headings that other pages link to are unchanged. The upgrade guide links to four of them, and the contributor guide links to the memory usage log section. Nothing linked to the removed section.
How are these changes tested?
Documentation only, no code paths touched.
I built the site locally with Sphinx before and after the change, and the change adds no warnings. MyST warns about any link to a missing heading anchor, and in the built HTML all eight links into this page's sections, from the upgrade guide and the contributor guide, reach an existing anchor. No page linked to the removed section.
npx prettier@latest --checkpasses on both pages.