Repository navigation
[fix](be) Treat non-positive topn expansion rates as unlimited #68246
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -59,7 +59,8 @@ struct AggregateFunctionTopNData { | |
| using DataType = typename PrimitiveTypeTraits<T>::CppType; | ||
| void set_paramenters(int input_top_num, int space_expand_rate = 50) { | ||
| top_num = input_top_num; | ||
| capacity = (uint64_t)top_num * space_expand_rate; | ||
| // Non-positive expansion rates retain all candidates during serialization and merging. | ||
| capacity = space_expand_rate <= 0 ? UINT64_MAX : (uint64_t)top_num * space_expand_rate; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Version the new TopN partial-state representation This changes a rate-zero partial from capacity
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Compatibility with mixed-version BEs and historical AGG_STATE payloads is outside the scope of this PR. We will not add execution-version gating or legacy TopN implementations for this change. Please evaluate the new semantics on a homogeneous upgraded cluster and do not treat this compatibility concern as a blocker for this PR. |
||
| } | ||
|
|
||
| void add(const StringRef& value, const UInt64& increment = 1) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| #include <gtest/gtest.h> | ||
|
|
||
| #include <cstdint> | ||
| #include <string> | ||
|
|
||
| #include "core/column/column_string.h" | ||
| #include "core/column/column_vector.h" | ||
| #include "core/string_buffer.hpp" | ||
| #include "exprs/aggregate/aggregate_function_topn.h" | ||
|
|
||
| namespace doris { | ||
| namespace { | ||
|
|
||
| template <PrimitiveType T> | ||
| AggregateFunctionTopNData<T> round_trip(const AggregateFunctionTopNData<T>& state) { | ||
| auto column = ColumnString::create(); | ||
| BufferWritable writer(*column); | ||
| state.write(writer); | ||
| writer.commit(); | ||
| BufferReadable reader(column->get_data_at(0)); | ||
| AggregateFunctionTopNData<T> result; | ||
| result.read(reader); | ||
| return result; | ||
| } | ||
|
|
||
| class AggregateFunctionTopNUnlimitedTest : public testing::TestWithParam<int> {}; | ||
|
|
||
| TEST_P(AggregateFunctionTopNUnlimitedTest, SerializeAndMergeStrings) { | ||
| AggregateFunctionTopNData<TYPE_STRING> lhs; | ||
| AggregateFunctionTopNData<TYPE_STRING> rhs; | ||
| lhs.set_paramenters(1, GetParam()); | ||
| rhs.set_paramenters(1, GetParam()); | ||
| // The global winner is not the most frequent value in either partial state. | ||
| lhs.add(std::string("a"), 3); | ||
| lhs.add(std::string("winner"), 2); | ||
| rhs.add(std::string("b"), 3); | ||
| rhs.add(std::string("winner"), 2); | ||
|
|
||
| auto partial = round_trip(lhs); | ||
| EXPECT_EQ(partial.counter_map, lhs.counter_map); | ||
| AggregateFunctionTopNData<TYPE_STRING> merged; | ||
| merged.merge(partial); | ||
| merged = round_trip(merged); | ||
| merged.merge(round_trip(rhs)); | ||
| merged = round_trip(merged); | ||
|
|
||
| ASSERT_EQ(merged.counter_map.size(), 3); | ||
| EXPECT_EQ(merged.counter_map.at("a"), 3); | ||
| EXPECT_EQ(merged.counter_map.at("b"), 3); | ||
| EXPECT_EQ(merged.counter_map.at("winner"), 4); | ||
| EXPECT_EQ(merged.get(), R"({"winner":4})"); | ||
|
|
||
| AggregateFunctionTopNData<TYPE_STRING> empty; | ||
| merged.merge(round_trip(empty)); | ||
| EXPECT_EQ(merged.get(), R"({"winner":4})"); | ||
|
|
||
| merged.reset(); | ||
| EXPECT_EQ(round_trip(merged).get(), "{}"); | ||
| merged.set_paramenters(1, GetParam()); | ||
| merged.add(std::string("new"), 7); | ||
| EXPECT_EQ(round_trip(merged).get(), R"({"new":7})"); | ||
| } | ||
|
|
||
| TEST_P(AggregateFunctionTopNUnlimitedTest, SerializeAndMergeWeightedIntegers) { | ||
| AggregateFunctionTopNData<TYPE_INT> lhs; | ||
| AggregateFunctionTopNData<TYPE_INT> rhs; | ||
| lhs.set_paramenters(2, GetParam()); | ||
| rhs.set_paramenters(2, GetParam()); | ||
| lhs.add(1, 10); | ||
| lhs.add(2, 7); | ||
| lhs.add(3, 6); | ||
| rhs.add(4, 11); | ||
| rhs.add(5, 8); | ||
| rhs.add(3, 6); | ||
|
|
||
| AggregateFunctionTopNData<TYPE_INT> merged; | ||
| merged.merge(round_trip(lhs)); | ||
| merged.merge(round_trip(rhs)); | ||
| merged = round_trip(merged); | ||
| ASSERT_EQ(merged.counter_map.size(), 5); | ||
| EXPECT_EQ(merged.counter_map.at(3), 12); | ||
| auto result = ColumnInt32::create(); | ||
| merged.insert_result_into(*result); | ||
| ASSERT_EQ(result->size(), 2); | ||
| EXPECT_EQ(result->get_element(0), 3); | ||
| EXPECT_EQ(result->get_element(1), 4); | ||
| } | ||
|
|
||
| INSTANTIATE_TEST_SUITE_P(NonPositiveRates, AggregateFunctionTopNUnlimitedTest, | ||
| testing::Values(0, -1, INT32_MIN)); | ||
|
|
||
| TEST(AggregateFunctionTopNTest, PositiveRateStillLimitsSerializedCandidates) { | ||
| AggregateFunctionTopNData<TYPE_INT> state; | ||
| state.set_paramenters(1, 2); | ||
| state.add(1, 3); | ||
| state.add(2, 2); | ||
| state.add(3, 1); | ||
| auto partial = round_trip(state); | ||
| ASSERT_EQ(partial.counter_map.size(), 2); | ||
| EXPECT_EQ(partial.counter_map.at(1), 3); | ||
| EXPECT_EQ(partial.counter_map.at(2), 2); | ||
|
|
||
| state.set_paramenters(1); | ||
| EXPECT_EQ(round_trip(state).counter_map, state.counter_map); | ||
| } | ||
|
|
||
| } // namespace | ||
| } // namespace doris |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| -- This file is automatically generated. You should know what you did if you want to edit this | ||
| -- !unlimited -- | ||
| {"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5] | ||
|
|
||
| -- !grouped -- | ||
| 1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1] | ||
| 2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4] | ||
| 3 \N \N \N \N \N | ||
|
|
||
| -- !empty -- | ||
| \N \N \N | ||
|
|
||
| -- !constant_input -- | ||
| {"a":2} ["a"] | ||
|
|
||
| -- !unlimited -- | ||
| {"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5] | ||
|
|
||
| -- !grouped -- | ||
| 1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1] | ||
| 2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4] | ||
| 3 \N \N \N \N \N | ||
|
|
||
| -- !empty -- | ||
| \N \N \N | ||
|
|
||
| -- !constant_input -- | ||
| {"a":2} ["a"] | ||
|
|
||
| -- !unlimited -- | ||
| {"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5] | ||
|
|
||
| -- !grouped -- | ||
| 1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1] | ||
| 2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4] | ||
| 3 \N \N \N \N \N | ||
|
|
||
| -- !empty -- | ||
| \N \N \N | ||
|
|
||
| -- !constant_input -- | ||
| {"a":2} ["a"] | ||
|
|
||
| -- !positive_and_default -- | ||
| {"a":3,"x":2} {"a":3,"x":2} [1, 4] [1, 4] [2, 5] [2, 5] | ||
|
|
||
| -- !unlimited -- | ||
| {"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5] | ||
|
|
||
| -- !grouped -- | ||
| 1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1] | ||
| 2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4] | ||
| 3 \N \N \N \N \N | ||
|
|
||
| -- !empty -- | ||
| \N \N \N | ||
|
|
||
| -- !constant_input -- | ||
| {"a":2} ["a"] | ||
|
|
||
| -- !unlimited -- | ||
| {"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5] | ||
|
|
||
| -- !grouped -- | ||
| 1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1] | ||
| 2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4] | ||
| 3 \N \N \N \N \N | ||
|
|
||
| -- !empty -- | ||
| \N \N \N | ||
|
|
||
| -- !constant_input -- | ||
| {"a":2} ["a"] | ||
|
|
||
| -- !unlimited -- | ||
| {"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5] | ||
|
|
||
| -- !grouped -- | ||
| 1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1] | ||
| 2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4] | ||
| 3 \N \N \N \N \N | ||
|
|
||
| -- !empty -- | ||
| \N \N \N | ||
|
|
||
| -- !constant_input -- | ||
| {"a":2} ["a"] | ||
|
|
||
| -- !positive_and_default -- | ||
| {"a":3,"x":2} {"a":3,"x":2} [1, 4] [1, 4] [2, 5] [2, 5] | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P2] Widen the indices before making rate zero unbounded
With
capacity = UINT64_MAX, a valid rate-zero state can accumulate more thanINT_MAXdistinct candidates because there is no cardinality ceiling. Direct finalization then castscounter_vector.size()tointbefore limiting it bytop_num, so a populated state can emit no values. If the state is serialized,write()also iterates the 64-bitelement_numberwithauto i = 0; theColumnStringsize check runs only incommit()afterwrite()returns, so it does not guard that signed overflow. Please use size-safe result bounds andsize_t/uint64_tloop indices.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The signed loop indices and the narrowing of counter_vector.size() to int already exist in the original implementation; the underlying integer-width issue was not introduced by this PR. Although the zero-rate path now retains all candidates, we will defer this pre-existing boundary issue and keep this PR focused on making non-positive expansion rates mean unlimited candidate capacity. Please do not treat this deferred issue as a blocker for this PR.