Skip to content

Polish better decoder (Testing) - #7428

Closed
zhuqi-lucas wants to merge 65 commits into
apache:mainfrom
zhuqi-lucas:polish_better_decoder
Closed

zhuqi-lucas wants to merge 65 commits into
apache:mainfrom
zhuqi-lucas:polish_better_decoder

Conversation

@zhuqi-lucas

@zhuqi-lucas zhuqi-lucas commented Apr 19, 2025 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Apr 19, 2025
@alamb

alamb commented Apr 29, 2025

Copy link
Copy Markdown
Contributor

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1013-gcp #13~24.04.1-Ubuntu SMP Wed Apr 2 16:34:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing polish_better_decoder (0c3aa9b) to cee5124 diff
BENCH_NAME=arrow_reader_row_filter
BENCH_COMMAND=cargo bench --all-features --bench arrow_reader_row_filter
BENCH_FILTER=
BENCH_BRANCH_NAME=polish_better_decoder
Results will be posted here when complete

@alamb

alamb commented Apr 29, 2025

Copy link
Copy Markdown
Contributor

🤖: Benchmark completed

Details

group                                                                                 main                                   polish_better_decoder
-----                                                                                 ----                                   ---------------------
arrow_reader_row_filter/Composite/all_columns/async                                   1.04      2.9±0.02ms        ? ?/sec    1.00      2.8±0.03ms        ? ?/sec
arrow_reader_row_filter/Composite/all_columns/sync                                    1.00      3.3±0.02ms        ? ?/sec    1.00      3.3±0.03ms        ? ?/sec
arrow_reader_row_filter/Composite/exclude_filter_column/async                         1.00      2.6±0.02ms        ? ?/sec    1.03      2.6±0.01ms        ? ?/sec
arrow_reader_row_filter/Composite/exclude_filter_column/sync                          1.00      2.8±0.02ms        ? ?/sec    1.00      2.8±0.01ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveClustered/all_columns/async                1.03      3.1±0.02ms        ? ?/sec    1.00      3.0±0.01ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveClustered/all_columns/sync                 1.00      3.3±0.02ms        ? ?/sec    1.01      3.3±0.02ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveClustered/exclude_filter_column/async      1.00      2.9±0.01ms        ? ?/sec    1.01      2.9±0.01ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveClustered/exclude_filter_column/sync       1.00      3.1±0.02ms        ? ?/sec    1.01      3.1±0.02ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveUnclustered/all_columns/async              1.00      6.5±0.04ms        ? ?/sec    1.00      6.5±0.03ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveUnclustered/all_columns/sync               1.00      6.5±0.02ms        ? ?/sec    1.03      6.7±0.06ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveUnclustered/exclude_filter_column/async    1.00      5.7±0.03ms        ? ?/sec    1.00      5.7±0.04ms        ? ?/sec
arrow_reader_row_filter/ModeratelySelectiveUnclustered/exclude_filter_column/sync     1.00      5.7±0.04ms        ? ?/sec    1.03      5.9±0.03ms        ? ?/sec
arrow_reader_row_filter/PointLookup/all_columns/async                                 1.00      2.1±0.01ms        ? ?/sec    1.00      2.1±0.01ms        ? ?/sec
arrow_reader_row_filter/PointLookup/all_columns/sync                                  1.00      2.2±0.01ms        ? ?/sec    1.01      2.2±0.01ms        ? ?/sec
arrow_reader_row_filter/PointLookup/exclude_filter_column/async                       1.00      2.0±0.01ms        ? ?/sec    1.00      2.0±0.01ms        ? ?/sec
arrow_reader_row_filter/PointLookup/exclude_filter_column/sync                        1.00      2.2±0.01ms        ? ?/sec    1.01      2.2±0.01ms        ? ?/sec
arrow_reader_row_filter/SelectiveUnclustered/all_columns/async                        1.02      3.2±0.03ms        ? ?/sec    1.00      3.1±0.03ms        ? ?/sec
arrow_reader_row_filter/SelectiveUnclustered/all_columns/sync                         1.00      3.4±0.04ms        ? ?/sec    1.01      3.4±0.02ms        ? ?/sec
arrow_reader_row_filter/SelectiveUnclustered/exclude_filter_column/async              1.00      2.8±0.01ms        ? ?/sec    1.03      2.9±0.02ms        ? ?/sec
arrow_reader_row_filter/SelectiveUnclustered/exclude_filter_column/sync               1.00      3.0±0.02ms        ? ?/sec    1.01      3.0±0.02ms        ? ?/sec
arrow_reader_row_filter/UnselectiveClustered/all_columns/async                        1.01      8.0±0.05ms        ? ?/sec    1.00      7.8±0.06ms        ? ?/sec
arrow_reader_row_filter/UnselectiveClustered/all_columns/sync                         1.01      8.3±0.07ms        ? ?/sec    1.00      8.1±0.07ms        ? ?/sec
arrow_reader_row_filter/UnselectiveClustered/exclude_filter_column/async              1.01      7.7±0.04ms        ? ?/sec    1.00      7.6±0.04ms        ? ?/sec
arrow_reader_row_filter/UnselectiveClustered/exclude_filter_column/sync               1.01      8.0±0.04ms        ? ?/sec    1.00      7.9±0.05ms        ? ?/sec
arrow_reader_row_filter/UnselectiveUnclustered/all_columns/async                      1.01      3.1±0.03ms        ? ?/sec    1.00      3.1±0.03ms        ? ?/sec
arrow_reader_row_filter/UnselectiveUnclustered/all_columns/sync                       1.00      3.4±0.01ms        ? ?/sec    1.02      3.5±0.03ms        ? ?/sec
arrow_reader_row_filter/UnselectiveUnclustered/exclude_filter_column/async            1.00      2.8±0.01ms        ? ?/sec    1.02      2.9±0.02ms        ? ?/sec
arrow_reader_row_filter/UnselectiveUnclustered/exclude_filter_column/sync             1.00      3.0±0.02ms        ? ?/sec    1.01      3.1±0.02ms        ? ?/sec
arrow_reader_row_filter/Utf8ViewNonEmpty/all_columns/async                            1.03     23.9±0.14ms        ? ?/sec    1.00     23.1±0.14ms        ? ?/sec
arrow_reader_row_filter/Utf8ViewNonEmpty/all_columns/sync                             1.00     23.9±0.13ms        ? ?/sec    1.00     23.8±0.12ms        ? ?/sec
arrow_reader_row_filter/Utf8ViewNonEmpty/exclude_filter_column/async                  1.00     15.2±0.06ms        ? ?/sec    1.00     15.2±0.11ms        ? ?/sec
arrow_reader_row_filter/Utf8ViewNonEmpty/exclude_filter_column/sync                   1.01     15.2±0.10ms        ? ?/sec    1.00     15.0±0.08ms        ? ?/sec

@alamb

alamb commented Apr 29, 2025

Copy link
Copy Markdown
Contributor

🤖: Benchmark completed

Interesting, the decoder cache doesn't seem to help much on my test machine (which is some crappy gcp VM). I couldn't reproduce the results listed on #7363 (comment) 🤔

@zhuqi-lucas

Copy link
Copy Markdown
Contributor Author

🤖: Benchmark completed

Interesting, the decoder cache doesn't seem to help much on my test machine (which is some crappy gcp VM). I couldn't reproduce the results listed on #7363 (comment) 🤔

Thank you @alamb , it seems no obvious improvement compares to main. This branch only improve PointLookup for 1000000 line big data set comparing to original better-decode.

I agree, we need to find how to mock clickbench result from arrow-rs side.

@alamb

alamb commented Apr 30, 2025

Copy link
Copy Markdown
Contributor

I agree, we need to find how to mock clickbench result from arrow-rs side.

@alamb alamb mentioned this pull request May 5, 2025
3 of 4 tasks
@github-actions

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actions github-actions Bot added the Stale label Jun 26, 2026
@github-actions github-actions Bot closed this Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate Stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants