Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion be/src/exprs/aggregate/aggregate_function_topn.h
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,8 @@ struct AggregateFunctionTopNData {
using DataType = typename PrimitiveTypeTraits<T>::CppType;
void set_paramenters(int input_top_num, int space_expand_rate = 50) {
top_num = input_top_num;
capacity = (uint64_t)top_num * space_expand_rate;
// Non-positive expansion rates retain all candidates during serialization and merging.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Widen the indices before making rate zero unbounded

With capacity = UINT64_MAX, a valid rate-zero state can accumulate more than INT_MAX distinct candidates because there is no cardinality ceiling. Direct finalization then casts counter_vector.size() to int before limiting it by top_num, so a populated state can emit no values. If the state is serialized, write() also iterates the 64-bit element_number with auto i = 0; the ColumnString size check runs only in commit() after write() returns, so it does not guard that signed overflow. Please use size-safe result bounds and size_t/uint64_t loop indices.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The signed loop indices and the narrowing of counter_vector.size() to int already exist in the original implementation; the underlying integer-width issue was not introduced by this PR. Although the zero-rate path now retains all candidates, we will defer this pre-existing boundary issue and keep this PR focused on making non-positive expansion rates mean unlimited candidate capacity. Please do not treat this deferred issue as a blocker for this PR.

capacity = space_expand_rate <= 0 ? UINT64_MAX : (uint64_t)top_num * space_expand_rate;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Version the new TopN partial-state representation

This changes a rate-zero partial from capacity 0 with zero serialized elements to capacity UINT64_MAX with every candidate, but all three TopN functions are still selected under the same BE execution version. During a rolling upgrade, an old BE therefore drops its rate-zero partition and the new merge() ignores that empty decoded map, so the result contains only rows scanned by upgraded BEs. Negative rates can fail instead: for N=2, rate=-1, an old partial carries capacity UINT64_MAX-1 and a new partial carries UINT64_MAX, triggering the incompatible-parameter check when both are nonempty. Stored AGG_STATE payloads are also still tagged version 15. Please introduce a new execution version and preserve/select the old TopN creators for version 15 (or otherwise fence mixed representations); reader-side normalization cannot recover candidates that the old rate-zero serializer omitted.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compatibility with mixed-version BEs and historical AGG_STATE payloads is outside the scope of this PR. We will not add execution-version gating or legacy TopN implementations for this change. Please evaluate the new semantics on a homogeneous upgraded cluster and do not treat this compatibility concern as a blocker for this PR.

}

void add(const StringRef& value, const UInt64& increment = 1) {
Expand Down
63 changes: 17 additions & 46 deletions be/test/exprs/aggregate/agg_state_parameters_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

#include <gtest/gtest.h>

#include <cstdint>
#include <limits>

#include "common/exception.h"
Expand Down Expand Up @@ -88,12 +89,6 @@ class StateParameterChecks {
});
}

void check_direct_mismatch(const Arguments& first, const Arguments& second) {
auto* destination = create(first);
auto* source = create(second);
expect_incompatible([&] { _function->merge(destination, source, _arena); });
}

void check_merge_result(const Arguments& initial, const Arguments& incoming, bool serialized) {
auto* destination = create();
auto* source = create();
Expand Down Expand Up @@ -147,21 +142,6 @@ class StateParameterChecks {
EXPECT_TRUE(ColumnHelper::column_equal(result(destination), result(expected)));
}

void check_configured_empty_payload(const Arguments& arguments) {
// TopN with zero capacity keeps counters in memory but serializes no counters.
// A decoded empty state must not change compatible counters.
auto serialized = serialize(create(arguments));
auto* configured_empty = create();
_function->deserialize_and_merge_from_column(configured_empty, *serialized, _arena);
auto* populated = create(arguments);
EXPECT_NO_THROW(
_function->deserialize_and_merge_from_column(populated, *serialized, _arena));
EXPECT_TRUE(ColumnHelper::column_equal(result(populated), result(create(arguments))));
EXPECT_NO_THROW(_function->merge(configured_empty, create(arguments), _arena));
EXPECT_TRUE(
ColumnHelper::column_equal(result(configured_empty), result(create(arguments))));
}

void check_compatible(const Arguments& arguments) {
auto* destination = create();
auto serialized = serialize(create(arguments));
Expand Down Expand Up @@ -394,33 +374,24 @@ TEST(AggregateStateParametersTest, Histogram) {
{value, argument<DataTypeInt32>(3)});
}

TEST(AggregateStateParametersTest, TopNZeroCapacityParameters) {
TEST(AggregateStateParametersTest, TopNUnlimitedParameters) {
for (const auto& name : {"topn", "topn_array", "topn_weighted"}) {
SCOPED_TRACE(name);
Arguments zero_capacity {argument<DataTypeString>("a")};
if (std::string(name) == "topn_weighted") {
zero_capacity.push_back(argument<DataTypeInt64>(1));
}
zero_capacity.push_back(argument<DataTypeInt32>(1));
zero_capacity.push_back(argument<DataTypeInt32>(0));
auto positive_capacity = zero_capacity;
positive_capacity.back() = argument<DataTypeInt32>(2);
DataTypes types;
for (const auto& arg : zero_capacity) {
types.push_back(arg.type);
for (int rate : {0, -1, INT32_MIN}) {
SCOPED_TRACE(rate);
Arguments unlimited {argument<DataTypeString>("a")};
if (std::string(name) == "topn_weighted") {
unlimited.push_back(argument<DataTypeInt64>(1));
}
unlimited.push_back(argument<DataTypeInt32>(1));
unlimited.push_back(argument<DataTypeInt32>(rate));
auto finite = unlimited;
finite.back() = argument<DataTypeInt32>(2);
check_parameters(name, unlimited, finite);

auto zero_rate = unlimited;
zero_rate.back() = argument<DataTypeInt32>(0);
check_compatible_states(name, unlimited, zero_rate);
}
auto function = AggregateFunctionSimpleFactory::instance().get(
name, types, nullptr, false, BeExecVersionManager::get_newest_version());
ASSERT_NE(function, nullptr);
StateParameterChecks checks(function);
// Only the serialized zero-capacity state is empty; in-memory counters contribute.
checks.check_direct_mismatch(zero_capacity, positive_capacity);
checks.check_direct_mismatch(positive_capacity, zero_capacity);
check_ignored_parameters(name, zero_capacity, positive_capacity, true);
checks.check_merge_result({}, zero_capacity, false);
checks.check_merge_result(zero_capacity, {}, false);
checks.check_merge_result(zero_capacity, {}, true);
checks.check_configured_empty_payload(zero_capacity);
}
}

Expand Down
125 changes: 125 additions & 0 deletions be/test/exprs/aggregate/agg_topn_test.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#include <gtest/gtest.h>

#include <cstdint>
#include <string>

#include "core/column/column_string.h"
#include "core/column/column_vector.h"
#include "core/string_buffer.hpp"
#include "exprs/aggregate/aggregate_function_topn.h"

namespace doris {
namespace {

template <PrimitiveType T>
AggregateFunctionTopNData<T> round_trip(const AggregateFunctionTopNData<T>& state) {
auto column = ColumnString::create();
BufferWritable writer(*column);
state.write(writer);
writer.commit();
BufferReadable reader(column->get_data_at(0));
AggregateFunctionTopNData<T> result;
result.read(reader);
return result;
}

class AggregateFunctionTopNUnlimitedTest : public testing::TestWithParam<int> {};

TEST_P(AggregateFunctionTopNUnlimitedTest, SerializeAndMergeStrings) {
AggregateFunctionTopNData<TYPE_STRING> lhs;
AggregateFunctionTopNData<TYPE_STRING> rhs;
lhs.set_paramenters(1, GetParam());
rhs.set_paramenters(1, GetParam());
// The global winner is not the most frequent value in either partial state.
lhs.add(std::string("a"), 3);
lhs.add(std::string("winner"), 2);
rhs.add(std::string("b"), 3);
rhs.add(std::string("winner"), 2);

auto partial = round_trip(lhs);
EXPECT_EQ(partial.counter_map, lhs.counter_map);
AggregateFunctionTopNData<TYPE_STRING> merged;
merged.merge(partial);
merged = round_trip(merged);
merged.merge(round_trip(rhs));
merged = round_trip(merged);

ASSERT_EQ(merged.counter_map.size(), 3);
EXPECT_EQ(merged.counter_map.at("a"), 3);
EXPECT_EQ(merged.counter_map.at("b"), 3);
EXPECT_EQ(merged.counter_map.at("winner"), 4);
EXPECT_EQ(merged.get(), R"({"winner":4})");

AggregateFunctionTopNData<TYPE_STRING> empty;
merged.merge(round_trip(empty));
EXPECT_EQ(merged.get(), R"({"winner":4})");

merged.reset();
EXPECT_EQ(round_trip(merged).get(), "{}");
merged.set_paramenters(1, GetParam());
merged.add(std::string("new"), 7);
EXPECT_EQ(round_trip(merged).get(), R"({"new":7})");
}

TEST_P(AggregateFunctionTopNUnlimitedTest, SerializeAndMergeWeightedIntegers) {
AggregateFunctionTopNData<TYPE_INT> lhs;
AggregateFunctionTopNData<TYPE_INT> rhs;
lhs.set_paramenters(2, GetParam());
rhs.set_paramenters(2, GetParam());
lhs.add(1, 10);
lhs.add(2, 7);
lhs.add(3, 6);
rhs.add(4, 11);
rhs.add(5, 8);
rhs.add(3, 6);

AggregateFunctionTopNData<TYPE_INT> merged;
merged.merge(round_trip(lhs));
merged.merge(round_trip(rhs));
merged = round_trip(merged);
ASSERT_EQ(merged.counter_map.size(), 5);
EXPECT_EQ(merged.counter_map.at(3), 12);
auto result = ColumnInt32::create();
merged.insert_result_into(*result);
ASSERT_EQ(result->size(), 2);
EXPECT_EQ(result->get_element(0), 3);
EXPECT_EQ(result->get_element(1), 4);
}

INSTANTIATE_TEST_SUITE_P(NonPositiveRates, AggregateFunctionTopNUnlimitedTest,
testing::Values(0, -1, INT32_MIN));

TEST(AggregateFunctionTopNTest, PositiveRateStillLimitsSerializedCandidates) {
AggregateFunctionTopNData<TYPE_INT> state;
state.set_paramenters(1, 2);
state.add(1, 3);
state.add(2, 2);
state.add(3, 1);
auto partial = round_trip(state);
ASSERT_EQ(partial.counter_map.size(), 2);
EXPECT_EQ(partial.counter_map.at(1), 3);
EXPECT_EQ(partial.counter_map.at(2), 2);

state.set_paramenters(1);
EXPECT_EQ(round_trip(state).counter_map, state.counter_map);
}

} // namespace
} // namespace doris
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !unlimited --
{"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5]

-- !grouped --
1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1]
2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4]
3 \N \N \N \N \N

-- !empty --
\N \N \N

-- !constant_input --
{"a":2} ["a"]

-- !unlimited --
{"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5]

-- !grouped --
1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1]
2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4]
3 \N \N \N \N \N

-- !empty --
\N \N \N

-- !constant_input --
{"a":2} ["a"]

-- !unlimited --
{"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5]

-- !grouped --
1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1]
2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4]
3 \N \N \N \N \N

-- !empty --
\N \N \N

-- !constant_input --
{"a":2} ["a"]

-- !positive_and_default --
{"a":3,"x":2} {"a":3,"x":2} [1, 4] [1, 4] [2, 5] [2, 5]

-- !unlimited --
{"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5]

-- !grouped --
1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1]
2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4]
3 \N \N \N \N \N

-- !empty --
\N \N \N

-- !constant_input --
{"a":2} ["a"]

-- !unlimited --
{"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5]

-- !grouped --
1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1]
2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4]
3 \N \N \N \N \N

-- !empty --
\N \N \N

-- !constant_input --
{"a":2} ["a"]

-- !unlimited --
{"a":3,"x":2} ["a", "x"] [1, 4] ["b", "y"] [2, 5]

-- !grouped --
1 {"a":3,"b":2} ["a", "b"] [1, 2] ["b", "a"] [2, 1]
2 {"x":2,"y":1} ["x", "y"] [4, 5] ["y", "x"] [5, 4]
3 \N \N \N \N \N

-- !empty --
\N \N \N

-- !constant_input --
{"a":2} ["a"]

-- !positive_and_default --
{"a":3,"x":2} {"a":3,"x":2} [1, 4] [1, 4] [2, 5] [2, 5]

Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,9 @@ suite("test_agg_state_parameters") {
def ignoredCases = [
["percentile_approx_weighted", "7, 0, 0.25", "7, 1, 0.75"],
["percentile_array", "7, cast([] as array<double>)", "7, [0.25]"],
["topn_weighted", "1, 1, 1, 0", "1, 1, 1, 2"],
["topn_array", "1, 1, 0", "1, 1, 2"],
["topn", "'a', 1, 0", "'a', 1, 2"]
["topn_weighted", "cast(NULL as int), 1, 1, 0", "nullable(cast(1 as int)), 1, 1, 2"],
["topn_array", "cast(NULL as int), 1, 0", "nullable(cast(1 as int)), 1, 2"],
["topn", "cast(NULL as string), 1, 0", "nullable(cast('a' as string)), 1, 2"]
]
def cases = [
["topn", "'a', 1", "'a', 3"],
Expand Down Expand Up @@ -105,6 +105,12 @@ suite("test_agg_state_parameters") {
["sequence_count", "'(?1)', non_nullable(cast('2024-01-01' as datetime)), true, false",
"'(?2)', non_nullable(cast('2024-01-01' as datetime)), true, false"]
]
// Unlimited states retain samples, so they cannot merge with finite-capacity states.
for (def rate : [0, -1, -2147483648]) {
cases.add(["topn", "'a', 1, ${rate}", "'a', 1, 2"])
cases.add(["topn_array", "1, 1, ${rate}", "1, 1, 2"])
cases.add(["topn_weighted", "1, 1, 1, ${rate}", "1, 1, 1, 2"])
}
// States without contributing samples ignore their parameter values.
// Keep AggState argument nullability identical across both sides of the UNION.
for (def quantile : ["0.0", "0.25", "1.0"]) {
Expand Down
Loading
Loading