Skip to content

branch-4.1: [fix](compaction) Seal index files after index compaction in non-vertical compaction - #69933

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
airborne12:fix-non-vertical-index-compaction-branch-4.1
Nov 4, 2026
Merged

yiguolei merged 2 commits into
apache:branch-4.1from
airborne12:fix-non-vertical-index-compaction-branch-4.1

Conversation

@airborne12

@airborne12 airborne12 commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #65858

Problem Summary:

With enable_vertical_compaction=false and inverted_index_compaction_enable=true, a DUP KEY or MoW table compacted through the non-vertical path loses the inverted indexes of its string columns.

  • Columns in columns_to_do_index_compaction are skipped by the segment writer and merged after all segments are flushed.
  • Non-vertical compaction flushes through SegmentFlusher, which called close_inverted_index() at flush time. With no logical index present, begin_close() sealed an empty container and uploaded a zero byte .idx.
  • Index compaction then wrote into temporary directories of the closed writer, and SegmentFlusher::close() only ran finish_close(), so the merged index was never packed. succeed to do index compaction is still logged.

Result: INDEX_DISK_SIZE=0, and each query logs [E-6004] inverted index file ... is empty and scans the whole segment. Query results stay correct.

With a numeric index on the same table, the file is sealed with the numeric index alone, so INDEX_DISK_SIZE stays above zero while the string index is gone.

master fixed this as part of the row TTL feature #65858. This PR carries only that part, without the feature: when the rowset has index compaction, the flush no longer closes the index files, and BaseBetaRowsetWriter::_close_file_writers() calls the existing _close_inverted_index_file_writers(), the same close the vertical writer uses. All segments are already flushed at that point, because the merger flushes the rowset writer when it finishes. This branch still has SegmentWriter, so both _flush_segment_writer overloads are changed.

A DCHECK in BlockReader::_direct_next_block is relaxed as well. At EOF the block is empty and the iterator still reports the previous block's locations, so ASAN and debug builds abort on this path. Merger only reads the locations when the block has rows, so release builds were not affected.

Existing rowsets are not repaired by this change.

Branch adaptations

  • has_index_compaction() checks only columns_to_do_index_compaction, because this branch has no SNII.
  • master seals in SegmentFlusher::close(). This PR reuses the rowset level close instead, so the result is the same but the code differs from master.
  • master covers this path with a BE unit test, not with a regression test. test_index_compaction_non_vertical is new here, and [improvement](compaction) Seal compaction index files through the rowset level close #69976 adds it to master with the same close.

Release note

Fix inverted indexes of string columns being lost by non-vertical compaction when inverted index compaction is enabled.

Check List (For Author)

  • Test
    • Regression test: in non-cloud mode on this branch, new test_index_compaction_non_vertical fails without the fix and passes with it. test_index_compaction_dup_keys, _unique_keys, _null and _empty_segments pass.
    • Unit Test
    • Manual test
    • No need to test or manual test.

Cloud mode on object storage was tested with the same change on a 4.1-based tree, not on this branch. BE unit tests were not run.

  • Behavior changed:

    • No.
    • Yes. INDEX_DISK_SIZE of such compaction output is now the real size and the output keeps its inverted indexes.
  • Does this need documentation?

    • No.
    • Yes.

🤖 Generated with Claude Code

…ical compaction

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65858

Problem Summary:

With enable_vertical_compaction=false and inverted_index_compaction_enable=true
(both mutable BE configs), a DUP KEY or MoW table compacted through the
non-vertical path loses the inverted indexes of its string columns.

Reproduce in cloud mode: create a DUP KEY table with an un-tokenized inverted
index on a TEXT column and storage format V2, load 6 batches, set
enable_vertical_compaction=false, run a cumulative compaction. The output
rowset has INDEX_DISK_SIZE=0, the log shows put_object for _0.idx with size=0
before "start index compaction", "succeed to do index compaction" is still
printed, and each query reports "[E-6004] inverted index file ... is empty"
and scans the whole segment.

Root cause: columns in columns_to_do_index_compaction are skipped by the
segment writer and merged by Compaction::do_inverted_index_compaction after
all segments are flushed. Non-vertical compaction flushes through
SegmentFlusher, which called close_inverted_index() at flush time. With no
logical index present, begin_close() sealed an empty container and uploaded a
zero byte object. Index compaction then wrote into temporary directories of
the already closed writer, and SegmentFlusher::close() only ran finish_close(),
so the merged index was never packed. With a numeric index on the same table
the file was sealed with the numeric index alone, so INDEX_DISK_SIZE stayed
above zero while the string index was gone. The vertical writer closes all
index files in _close_file_writers() and is not affected.

Fix, same as the change made on master: when the rowset has index compaction
columns, defer sealing and size accounting to SegmentFlusher::close(), and add
the collected size to the rowset total in _close_file_writers().

Also relax a DCHECK in BlockReader::_direct_next_block. At EOF the block is
empty and the iterator still reports the locations of the previous block, so
ASAN and debug builds abort on this path. Merger only reads the locations when
the block has rows, so release builds were not affected.

Existing rowsets are not repaired by this change. A rowset is rebuilt when it
is compacted together with other rowsets, or by DROP INDEX and ADD INDEX.

### Release note

Fix inverted indexes of string columns being lost by non-vertical compaction
when inverted index compaction is enabled.

### Check List (For Author)

- Test: Regression test
    - In non-cloud mode on this branch, new suite
      test_index_compaction_non_vertical fails without the fix and passes with
      it; test_index_compaction_dup_keys, _unique_keys, _null and
      _empty_segments pass
    - Cloud mode on object storage was tested with the same change on a
      4.1-based tree, not on this branch
    - BE unit tests not run
- Behavior changed: Yes. INDEX_DISK_SIZE of such compaction output is now the
  real size and the output keeps its inverted indexes
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@airborne12
airborne12 requested a review from yiguolei as a code owner November 2, 2026 00:19
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

… compaction

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65858

Problem Summary:

The first commit sealed the index files in SegmentFlusher::close() and added
their size in BaseBetaRowsetWriter::_close_file_writers(), which repeats the
sequence already in _close_inverted_index_file_writers(). Call that helper from
_close_file_writers() when the rowset has index compaction, as the vertical
writer does. Sealing, size accounting and waiting now stay in one place and
SegmentFlusher::close() is unchanged.

All segments are flushed before index compaction starts, because the merger
flushes the rowset writer when it finishes. SegmentFlusher::close() still runs
finish_close() afterwards, which does nothing for a closed writer.

### Release note

None

### Check List (For Author)

- Test: Regression test, see the PR description
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@airborne12

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.15% (32287/42962)
Line Coverage 59.59% (362203/607849)
Region Coverage 56.18% (301416/536529)
Branch Coverage 57.18% (137387/240260)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 81.82% (9/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.17% (32296/42962)
Line Coverage 59.63% (362446/607848)
Region Coverage 56.20% (301553/536529)
Branch Coverage 57.23% (137506/240260)

@yiguolei
yiguolei merged commit 0789d53 into apache:branch-4.1 Nov 4, 2026
42 of 43 checks passed
@airborne12
airborne12 deleted the fix-non-vertical-index-compaction-branch-4.1 branch November 4, 2026 11:05
yiguolei pushed a commit that referenced this pull request Nov 6, 2026
…set level close (#69976)

### What problem does this PR solve?

Issue Number: None

Related PR: #65858

Problem Summary:

#65858 fixed non-vertical compaction losing string indexes, but the fix
made the close path depend on one feature:

- `SegmentFlusher::close()` asked `has_index_compaction()` before
sealing.
- `BaseBetaRowsetWriter::_close_file_writers()` added a collection total
that is only meaningful for deferred writers, and relied on it being
zero for everything else.
- The sequence of sealing, adding the size and waiting already existed
in `_close_inverted_index_file_writers()`, which the vertical writer
uses.

Closing does not need to know about index compaction. It only has to
seal the index files that are still open.

This PR makes three changes:

1. **Closing no longer checks for index compaction.**
`_close_file_writers()` always calls
`_close_inverted_index_file_writers()` after the segments are closed.
`InvertedIndexFileCollection::begin_close()` skips writers sealed when
their segment was flushed, and `SegmentFlusher::close()` only closes
segments. The one remaining `has_index_compaction()` check is at flush,
where the flusher has to know that postings are still to come.
2. **`IndexFileWriter::open()` rejects a closed writer.** The original
bug stayed silent because `open()` accepted a closed writer and the
directory it returned never reached the sealed file. A mistake of this
kind now fails the compaction instead of committing a rowset without its
index.
3. **End to end test.** `test_index_compaction_non_vertical` covers this
path, including a table with both numeric and string indexes. The
existing BE unit test
`HorizontalCompactionSealsIndexesAfterPostingsAreWritten` is unchanged
and passes.

Index size accounting is unchanged:

- A writer sealed at flush is counted in the segment statistics and
skipped by the collection, so it is not added twice.
- A deferred writer is added by the helper, the same `_total_index_size`
update as before.

`finish_close()` keeps its position after the segment files are closed
and the preload is done, so loads and compactions without index
compaction close in the same order as before.

Backports: #69933 (branch-4.1) and #69936 (branch-4.2) carry the fix
with a check in `_close_file_writers()`. They do not have this cleanup
yet.

### Release note

None

### Check List (For Author)

- Test
- [x] Regression test: in non-cloud mode, all 7 suites in
`inverted_index_p0/index_compaction` pass, including the new
`test_index_compaction_non_vertical`.
- [x] Unit Test: 120 tests pass, covering `IndexFileWriterTest` with the
new `OpenAfterCloseFails` and `CollectionClosesOnlyOpenWriters`,
`SegmentFlusherTransformFormatTest`, segment compaction,
`BetaRowsetTest` and cloud compaction.
- [x] Manual test: full compaction with vertical off on V2, V3 and SNII,
with string indexes only and with numeric plus string indexes. Index
size is above zero, matches the previous commit, and no index downgrade
happens.
    - [ ] No need to test or manual test.

Cloud mode was not run for this change.

- Behavior changed:
    - [ ] No.
- [x] Yes. Opening an index on a closed `IndexFileWriter` now returns an
error. No such caller exists after this change.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: martin.foo838 <martin.foo838@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants