Repository navigation
branch-4.2: [fix](compaction) Seal index files after index compaction in non-vertical compaction - #69936
Merged
yiguolei merged 2 commits intoNov 3, 2026
Conversation
…ical compaction ### What problem does this PR solve? Issue Number: None Related PR: apache#65858 Problem Summary: With enable_vertical_compaction=false and inverted_index_compaction_enable=true (both mutable BE configs), a DUP KEY or MoW table compacted through the non-vertical path loses the inverted indexes of its string columns. Reproduce in cloud mode: create a DUP KEY table with an un-tokenized inverted index on a TEXT column and storage format V2, load 6 batches, set enable_vertical_compaction=false, run a cumulative compaction. The output rowset has INDEX_DISK_SIZE=0, the log shows put_object for _0.idx with size=0 before "start index compaction", "succeed to do index compaction" is still printed, and each query reports "[E-6004] inverted index file ... is empty" and scans the whole segment. Root cause: columns in columns_to_do_index_compaction are skipped by the segment writer and merged by Compaction::do_inverted_index_compaction after all segments are flushed. Non-vertical compaction flushes through SegmentFlusher, which called close_inverted_index() at flush time. With no logical index present, begin_close() sealed an empty container and uploaded a zero byte object. Index compaction then wrote into temporary directories of the already closed writer, and SegmentFlusher::close() only ran finish_close(), so the merged index was never packed. With a numeric index on the same table the file was sealed with the numeric index alone, so INDEX_DISK_SIZE stayed above zero while the string index was gone. The vertical writer closes all index files in _close_file_writers() and is not affected. Fix, same as the change made on master: when the rowset has index compaction columns, defer sealing and size accounting to SegmentFlusher::close(), and add the collected size to the rowset total in _close_file_writers(). Also relax a DCHECK in BlockReader::_direct_next_block. At EOF the block is empty and the iterator still reports the locations of the previous block, so ASAN and debug builds abort on this path. Merger only reads the locations when the block has rows, so release builds were not affected. Existing rowsets are not repaired by this change. A rowset is rebuilt when it is compacted together with other rowsets, or by DROP INDEX and ADD INDEX. ### Release note Fix inverted indexes of string columns being lost by non-vertical compaction when inverted index compaction is enabled. ### Check List (For Author) - Test: Regression test, Manual test - In non-cloud mode on this branch, new suite test_index_compaction_non_vertical and test_index_compaction_dup_keys, _unique_keys, _null and _empty_segments pass - Manual full compaction with vertical off on V2, V3 and SNII, with string indexes only and with numeric plus string indexes: index size above zero and no index downgrade - The failure without the fix was checked on branch-4.1, not on this branch - Cloud mode and BE unit tests not run on this branch - Behavior changed: Yes. INDEX_DISK_SIZE of such compaction output is now the real size and the output keeps its inverted indexes - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
… compaction ### What problem does this PR solve? Issue Number: None Related PR: apache#65858 Problem Summary: The first commit sealed the index files in SegmentFlusher::close() and added their size in BaseBetaRowsetWriter::_close_file_writers(), which repeats the sequence already in _close_inverted_index_file_writers(). Call that helper from _close_file_writers() when the rowset has index compaction, as the vertical writer does. Sealing, size accounting and waiting now stay in one place and SegmentFlusher::close() is unchanged. All segments are flushed before index compaction starts, because the merger flushes the rowset writer when it finishes. SegmentFlusher::close() still runs finish_close() afterwards, which does nothing for a closed writer. ### Release note None ### Check List (For Author) - Test: Regression test, see the PR description - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
run buildall |
5 of 8 tasks
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Nov 3, 2026
yiguolei
pushed a commit
that referenced
this pull request
Nov 6, 2026
…set level close (#69976) ### What problem does this PR solve? Issue Number: None Related PR: #65858 Problem Summary: #65858 fixed non-vertical compaction losing string indexes, but the fix made the close path depend on one feature: - `SegmentFlusher::close()` asked `has_index_compaction()` before sealing. - `BaseBetaRowsetWriter::_close_file_writers()` added a collection total that is only meaningful for deferred writers, and relied on it being zero for everything else. - The sequence of sealing, adding the size and waiting already existed in `_close_inverted_index_file_writers()`, which the vertical writer uses. Closing does not need to know about index compaction. It only has to seal the index files that are still open. This PR makes three changes: 1. **Closing no longer checks for index compaction.** `_close_file_writers()` always calls `_close_inverted_index_file_writers()` after the segments are closed. `InvertedIndexFileCollection::begin_close()` skips writers sealed when their segment was flushed, and `SegmentFlusher::close()` only closes segments. The one remaining `has_index_compaction()` check is at flush, where the flusher has to know that postings are still to come. 2. **`IndexFileWriter::open()` rejects a closed writer.** The original bug stayed silent because `open()` accepted a closed writer and the directory it returned never reached the sealed file. A mistake of this kind now fails the compaction instead of committing a rowset without its index. 3. **End to end test.** `test_index_compaction_non_vertical` covers this path, including a table with both numeric and string indexes. The existing BE unit test `HorizontalCompactionSealsIndexesAfterPostingsAreWritten` is unchanged and passes. Index size accounting is unchanged: - A writer sealed at flush is counted in the segment statistics and skipped by the collection, so it is not added twice. - A deferred writer is added by the helper, the same `_total_index_size` update as before. `finish_close()` keeps its position after the segment files are closed and the preload is done, so loads and compactions without index compaction close in the same order as before. Backports: #69933 (branch-4.1) and #69936 (branch-4.2) carry the fix with a check in `_close_file_writers()`. They do not have this cleanup yet. ### Release note None ### Check List (For Author) - Test - [x] Regression test: in non-cloud mode, all 7 suites in `inverted_index_p0/index_compaction` pass, including the new `test_index_compaction_non_vertical`. - [x] Unit Test: 120 tests pass, covering `IndexFileWriterTest` with the new `OpenAfterCloseFails` and `CollectionClosesOnlyOpenWriters`, `SegmentFlusherTransformFormatTest`, segment compaction, `BetaRowsetTest` and cloud compaction. - [x] Manual test: full compaction with vertical off on V2, V3 and SNII, with string indexes only and with numeric plus string indexes. Index size is above zero, matches the previous commit, and no index downgrade happens. - [ ] No need to test or manual test. Cloud mode was not run for this change. - Behavior changed: - [ ] No. - [x] Yes. Opening an index on a closed `IndexFileWriter` now returns an error. No such caller exists after this change. - Does this need documentation? - [x] No. - [ ] Yes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: martin.foo838 <martin.foo838@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #65858
Problem Summary:
With
enable_vertical_compaction=falseandinverted_index_compaction_enable=true, a DUP KEY or MoW table compacted through the non-vertical path loses the inverted indexes of its string columns.columns_to_do_index_compactionare skipped by the segment writer and merged after all segments are flushed.SegmentFlusher, which calledclose_inverted_index()at flush time. With no logical index present,begin_close()sealed an empty container and uploaded a zero byte.idx.SegmentFlusher::close()only ranfinish_close(), so the merged index was never packed.succeed to do index compactionis still logged.Result:
INDEX_DISK_SIZE=0, and each query logs[E-6004] inverted index file ... is emptyand scans the whole segment. Query results stay correct.With a numeric index on the same table, the file is sealed with the numeric index alone, so
INDEX_DISK_SIZEstays above zero while the string index is gone.master fixed this as part of the row TTL feature #65858. This PR carries only that part, without the feature: when the rowset has index compaction, the flush no longer closes the index files, and
BaseBetaRowsetWriter::_close_file_writers()calls the existing_close_inverted_index_file_writers(), the same close the vertical writer uses. All segments are already flushed at that point, because the merger flushes the rowset writer when it finishes. This branch still hasSegmentWriter, so both_flush_segment_writeroverloads are changed.A
DCHECKinBlockReader::_direct_next_blockis relaxed as well. At EOF the block is empty and the iterator still reports the previous block's locations, so ASAN and debug builds abort on this path.Mergeronly reads the locations when the block has rows, so release builds were not affected.Existing rowsets are not repaired by this change.
Branch adaptations
has_index_compaction()also checkssnii_indexes_to_do_compaction, same as master.SegmentFlusher::close(). This PR reuses the rowset level close instead, so the result is the same but the code differs from master.test_index_compaction_non_verticalis new here, and [improvement](compaction) Seal compaction index files through the rowset level close #69976 adds it to master with the same close.Release note
Fix inverted indexes of string columns being lost by non-vertical compaction when inverted index compaction is enabled.
Check List (For Author)
test_index_compaction_non_verticalandtest_index_compaction_dup_keys,_unique_keys,_nulland_empty_segmentspass.The failure without the fix was checked on branch-4.1, not on this branch. Cloud mode and BE unit tests were not run on this branch. The local build disabled one lance call because the installed lance-c is older than this branch needs; that change is not in this PR.
Behavior changed:
INDEX_DISK_SIZEof such compaction output is now the real size and the output keeps its inverted indexes.Does this need documentation?
🤖 Generated with Claude Code