Skip to content

Investigate why Utf8View performs worse for sort with high cardinality cases and improve it. #15402

Open

Description

@zhuqi-lucas

Is your feature request related to a problem or challenge?

This is a follow-up for the comments:
#15348 (comment)

Original result:

High Cardinality Performance

Sorting Method utf8 Time (ms) utf8_view Time (ms) utf8_view Improvement
merge sorted 4.6662 5.0999 -9.3% (slower)
sort merge 4.7102 5.7224 -21.5% (slower)
sort 7.0020 6.3274 9.6% faster
sort partitioned 242.99 µs 679.86 µs -180% (much slower)

Observations

  • utf8_view performs worse for high cardinality cases:
    • merge sorted is 9.3% slower.
    • sort merge is 21.5% slower.
    • sort partitioned is 180% slower, a drastic drop.
  • However, utf8_view still improves the sort method by 9.6%, likely due to reduced string operations.

We need to investigate why Utf8View performs worse for high cardinality cases and improve it.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. changed the title [-]Investigate why Utf8View performs worse for high cardinality cases and improve it.[/-] [+]Investigate why Utf8View performs worse for sort with high cardinality cases and improve it.[/+] on Mar 25, 2025
  2. zhuqi-lucas commented on Mar 25, 2025

    @zhuqi-lucas
    ContributorAuthor

    Latest testing result:

    Benchmark utf8 Time utf8_view Time Change (%) Performance
    merge sorted utf8 low cardinality 3.0828 ms 2.5903 ms -16.0% Improved
    sort merge utf8 low cardinality 3.1103 ms 2.6354 ms -15.3% Improved
    sort utf8 low cardinality 4.5160 ms 3.9261 ms -13.1% Improved
    sort partitioned utf8 low cardinality 193.02 µs 190.20 µs -1.5% Improved
    merge sorted utf8 high cardinality 4.5441 ms 3.9072 ms -14.0% Improved
    sort merge utf8 high cardinality 4.6552 ms 4.2851 ms -8.0% Improved
    sort utf8 high cardinality 5.3901 ms 4.5989 ms -14.7% Improved
    sort partitioned utf8 high cardinality 223.88 µs 2.6200 ms +1070.0% Regressed
    merge sorted utf8 tuple 10.669 ms 13.209 ms +23.8% Regressed
    sort merge utf8 tuple 13.204 ms 15.009 ms +13.7% Regressed
    sort utf8 tuple 10.280 ms 10.098 ms -1.8% Improved
    sort partitioned utf8 tuple 3.2097 ms 3.5555 ms +10.8% Regressed
    merge sorted mixed tuple 9.2576 ms 10.770 ms +16.3% Regressed
    sort merge mixed tuple 10.520 ms 11.871 ms +12.8% Regressed
    sort mixed tuple 9.4906 ms 8.7883 ms -7.4% Improved
    sort partitioned mixed tuple 2.4459 ms 3.2249 ms +31.9% Regressed
  3. LSUDOKO commented on Mar 28, 2025

    @LSUDOKO

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions