Skip to content

[test](regression) Pin row counts in bucket shuffle set operation tests - #70546

Open
924060929 wants to merge 1 commit into
apache:masterfrom
924060929:codex/stabilize-bucket-shuffle-set-operation
Open

924060929 wants to merge 1 commit into
apache:masterfrom
924060929:codex/stabilize-bucket-shuffle-set-operation

Conversation

@924060929

Copy link
Copy Markdown
Contributor

Background

The bucket-shuffle set-operation suite checks physical plan shapes as well as query results. INTERSECT input ordering and bucket-shuffle anchor selection depend on estimated row counts.

What problem does this PR solve?

Issue Number: None
Related PR: #65129 (enabled the suite and added bucket-shuffle coverage)
PR Type: Test

Problem Summary:

The three initial tables each contain three rows, but asynchronous tablet statistics updates can leave one estimated at three rows and another at one row. This changes INTERSECT input order or the UNION anchor and causes intermittent shape mismatches. Inject row_count=3 for both columns of these tables before checking plans. Keep the existing queries, assertions, and golden results.

Details

  • Failure: INTERSECT children can swap, or the join above UNION ALL can change from colocated to bucket shuffle.
  • Cause: The plan checks relied on asynchronously reported row counts immediately after creating and loading the tables.
  • Stability evidence: The complete suite passed 10 consecutive runs with default settings and another 10 with enable_spill=true. Only row counts are specified: adding detailed column statistics changes an unrelated join choice and bypasses the parent-hash-request coverage. All existing shape and result checks remain intact, including the fixtures with larger injected row counts later in the suite.

Testing

On a local macOS instance with one FE and one ASAN BE, built from b325dcd:

  • Single complete suite run passed.
  • ./run-regression-test.sh --run -d query_p0/set_operations -s bucket_shuffle_set_operation -parallel 1 -times 10: 10/10 passed with default settings.
  • The same command with global enable_spill=true: 10/10 passed.
  • Regression framework compilation and its 50 unit tests passed.
  • FE build passed with zero Checkstyle violations; BE build passed.
  • git diff --check passed. No golden output changes.

Release note

None

Check List (For Author)

  • Behavior changed: No
  • Does this need documentation: No
  • Pre-submission status: Ready

AI Disclosure

  • Tool: Codex
  • Generated: Test setup and PR description.
  • Human review: Pending maintainer review.

@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70546 at head ad96168 found no new findings. The only change adds six fixed-name ALTER TABLE ... MODIFY COLUMN ... SET STATS ('row_count'='3') statements after creating and loading three-row OLAP fixtures. Source tracing confirms each statement validates synchronously, writes and synchronizes the column statistic, marks the base-index table row count as user-injected, and is consumed by Nereids when selecting the bucket-shuffle basic child. The injected count matches the data, and count-only stats leave NDV untrustable, so no result or unrelated join-plan coverage defect was substantiated. The change is focused and follows existing regression-test syntax and setup conventions; no concurrency, lifecycle, configuration, protocol, persistence, or source-data path is introduced. Existing inline threads: none. User review focus: none. Static validation covered the complete authoritative diff, changed-path list, existing review context, and upstream/downstream statistics flow; no builds or runtime tests were executed under the static-review contract. Review converged in round 1 because both assigned review passes returned NO_NEW_VALUABLE_FINDINGS and the only risk focus was dismissed with evidence.

Review converged after 1 round(s).

@924060929

Copy link
Copy Markdown
Contributor Author

run compile

### Background

The bucket-shuffle set-operation suite checks physical plan shapes as well as query results. INTERSECT input ordering and bucket-shuffle anchor selection depend on estimated row counts.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#65129 (enabled the suite and added bucket-shuffle coverage)
PR Type: Test

Problem Summary:

The three initial tables each contain three rows, but asynchronous tablet statistics updates can leave one estimated at three rows and another at one row. This changes INTERSECT input order or the UNION anchor and causes intermittent shape mismatches. Inject row_count=3 for both columns of these tables before checking plans. Keep the existing queries, assertions, and golden results.

### Details

- **Failure:** INTERSECT children can swap, or the join above UNION ALL can change from colocated to bucket shuffle.
- **Cause:** The plan checks relied on asynchronously reported row counts immediately after creating and loading the tables.
- **Stability evidence:** The complete suite passed 10 consecutive runs with default settings and another 10 with enable_spill=true. Only row counts are specified: adding detailed column statistics changes an unrelated join choice and bypasses the parent-hash-request coverage. All existing shape and result checks remain intact, including the fixtures with larger injected row counts later in the suite.

### Testing

On a local macOS instance with one FE and one ASAN BE, built from b325dcd:

- Single complete suite run passed.
- `./run-regression-test.sh --run -d query_p0/set_operations -s bucket_shuffle_set_operation -parallel 1 -times 10`: 10/10 passed with default settings.
- The same command with global enable_spill=true: 10/10 passed.
- Regression framework compilation and its 50 unit tests passed.
- FE build passed with zero Checkstyle violations; BE build passed.
- `git diff --check` passed. No golden output changes.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex
- Generated: Test setup and PR description.
- Human review: Pending maintainer review.
@924060929
924060929 force-pushed the codex/stabilize-bucket-shuffle-set-operation branch from ad96168 to af151e9 Compare November 16, 2026 06:33
@924060929

Copy link
Copy Markdown
Contributor Author

run buildall

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/43817566159

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review setup stopped before context preparation ran; inspect the first failed or cancelled setup step.
Workflow run: https://github.com/apache/doris/actions/runs/44015548400

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@924060929

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70546 at af151e9 found no new findings.

The change pins the three initial fixtures at their actual three-row cardinalities before any plan or result check. Source tracing confirms that ALTER stats publishes user-injected base-index counts, which Nereids uses ahead of tablet-reported counts. The incomplete column records become UNKNOWN through ColStatsData.isValid(), preserving unknown detailed column statistics rather than introducing false NDV/range estimates. Cache reloads apply the same validation.

Critical checkpoint conclusions:

  1. Goal and proof: the setup addresses the identified row-count race. Existing shape assertions and ordered-result goldens remain intact; runtime stability was not independently tested here.
  2. Scope and clarity: ten setup lines in one regression file, using the existing statistics-injection facility.
  3. Concurrency and locks: SQL statements complete sequentially. Injected counts take precedence over asynchronous tablet reports; no threads, locks or lock-order changes are introduced.
  4. Lifecycle: fixtures are recreated before use, statistics are keyed by table IDs, and the suite reuses its connection. ForwardWithSync waits for metadata replay on a follower; no new resource lifetime is introduced.
  5. Configuration: no configuration item or session-variable assignment is added.
  6. Compatibility: no function symbols, storage formats or production behavior change.
  7. Thrift/protobuf: no gensrc changes; field-ID comparisons are inapplicable.
  8. Parallel paths: all three initial fixtures and both columns are covered. Id-only, value-only, full-row, unequal-bucket, local-shuffle and planner-switch cases remain present. Later right-basic and fill-up tables retain their independent larger counts.
  9. Conditional checks: no new conditional or fallback; fixed loops cover the intended six table/column pairs.
  10. Test coverage: the complete suite was inspected, including UNION/INTERSECT/EXCEPT, joins, windows, parent-hash requests, right-basic selection, missing-bucket/equivalent-key cases, multi-column fallback and nullable keys. No new kernel feature requires unit tests.
  11. Test results: no golden changes. All existing output blocks were inspected against unchanged fixture data and queries; no executed result is claimed.
  12. Observability: existing SQL logging and explain/result failure diagnostics cover this setup addition; no production metrics are needed.
  13. Transactions and persistence: existing statistics persistence, metadata journaling and forwarding are reused without implementation or failover-protocol changes.
  14. Data writes and atomicity: statistics DDL follows completed fixture inserts. Source rows are unchanged; SQL errors propagate and fail the suite.
  15. FE/BE variables: no transmitted variable or protocol field is added.
  16. Performance: six bounded metadata statements per suite; no production hot path, large allocation or unbounded work is added.
  17. Other issues and core invariants: no change to data visibility, visible versions, delete bitmaps, memory ownership or nullability. No further defect was substantiated after rechecking column-cache validation and downstream optimizer consumers.

Existing inline threads and independently confirmed P0/P1 blockers: none. Additional user review focus: none.

Review converged in round 1: both the complete-review agent and the separate statistics-risk agent returned NO_NEW_VALUABLE_FINDINGS. Main-agent verification dismissed both risk focuses with evidence; no unresolved candidates or uncovered changed paths remain. Validation was static inspection only: no builds, tests or benchmarks ran, and the author's repeated-run claims were not independently reproduced.

Review converged after 1 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant