Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion be/src/common/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1228,7 +1228,7 @@ DEFINE_Validator(variant_storage_parse_mode,
[](const int config) -> bool { return config >= 0 && config <= 2; });

// block file cache
DEFINE_Bool(enable_file_cache, "false");
DEFINE_Bool(enable_file_cache, "true");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Ensure the default cache has a usable directory before enabling it. A coupled BE can have writable storage_root_path=/data/doris but read-only ${DORIS_HOME}; the default cache path ${DORIS_HOME}/file_cache then fails creation and this new true default makes BE startup fail. With ignore_broken_disk=true, the failed sole cache is skipped and startup succeeds with _caches empty; the first cached read or Rowset::clear_cache() calls get_by_path(hash), which takes % _caches.size() and crashes. Please provide a usable default/fallback and reject or disable caching when every path is skipped.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, two parts:

  • Empty factory: fixed in 747dd54. create_file_caches now fails when every configured path was skipped, so startup stops with "failed to init file cache ... All N file cache paths are broken", the same way doris_main handles storage and spill paths ("All disks are broken, exit."). The crash would in fact come before the first cached read: Rowset::clear_cache() looks up the cache for every removed rowset while enable_file_cache is on. Covered by the new BlockFileCacheTest.create_file_caches_rejects_a_factory_left_without_any_cache.
  • Unusable default directory: failing to start is intended. branch-4.1 (from 4.1.5) and branch-4.2 already enable the cache by default with the same path and the same behavior. Silently falling back to another disk would put cache data where the operator did not ask for it. The release note now states that BE refuses to start when ${DORIS_HOME}/file_cache cannot be created, and how to set file_cache_path or enable_file_cache = false.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Validate the cache block size before enabling it by default. A coupled BE that previously ran with enable_file_cache=false and file_cache_each_block_size=0 now enters ExecEnv::init_file_cache_factory(), where s3_write_buffer_size % file_cache_each_block_size executes before validate_hdfs_write_batch_buffer_size() rejects nonpositive sizes. Startup terminates with SIGFPE instead of reporting the invalid setting. Move the positive-size check ahead of the modulo.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0b2a2f2. init_file_cache_factory now runs validate_hdfs_write_batch_buffer_size first. It rejects a nonpositive file_cache_each_block_size ("file_cache_each_block_size 0 must be positive"), so the s3_write_buffer_size % file_cache_each_block_size check below can no longer divide by zero. Valid configurations go through the same checks as before, only in a different order. HdfsFileSystemTest.RejectsCacheBlockSizeNotDividingHdfsBatch now also covers block sizes 0 and -1.

// ATTENTION: For test only. Keep this enabled in production.
// Whether S3 storage write paths populate file cache while writing data to object storage.
// Disable this for tests that need load and compaction output to bypass file cache while keeping
Expand Down
6 changes: 6 additions & 0 deletions be/src/io/cache/block_file_cache_factory.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,12 @@ Status FileCacheFactory::create_file_caches(
return result.status;
}
}
// get_by_path() picks an instance by hashing modulo the instance count, so an empty factory
// would crash the first cached read or rowset cleanup. Refuse to start instead, as doris_main
// does when every storage or spill path is broken.
if (std::ranges::all_of(results, [](const BuildResult& result) { return result.skip; })) {
return Status::InternalError("All {} file cache paths are broken", results.size());
}

{
std::lock_guard lock(_mtx);
Expand Down
14 changes: 8 additions & 6 deletions be/src/runtime/exec_env_init.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -511,6 +511,14 @@ void ExecEnv::init_file_cache_factory(std::vector<doris::CachePath>& cache_paths
}
return;
}
// Runs first because it rejects a nonpositive file_cache_each_block_size, which the
// s3_write_buffer_size check below divides by.
auto hdfs_batch_status = io::validate_hdfs_write_batch_buffer_size(
config::hdfs_write_batch_buffer_size_mb, config::file_cache_each_block_size);
if (!hdfs_batch_status.ok()) {
LOG_FATAL("{}", hdfs_batch_status.to_string());
exit(-1);
}
if (config::file_cache_each_block_size > config::s3_write_buffer_size ||
config::s3_write_buffer_size % config::file_cache_each_block_size != 0) {
LOG_FATAL(
Expand All @@ -520,12 +528,6 @@ void ExecEnv::init_file_cache_factory(std::vector<doris::CachePath>& cache_paths
config::file_cache_each_block_size, config::s3_write_buffer_size);
exit(-1);
}
auto hdfs_batch_status = io::validate_hdfs_write_batch_buffer_size(
config::hdfs_write_batch_buffer_size_mb, config::file_cache_each_block_size);
if (!hdfs_batch_status.ok()) {
LOG_FATAL("{}", hdfs_batch_status.to_string());
exit(-1);
}
Status rest = doris::parse_conf_cache_paths(doris::config::file_cache_path, cache_paths);
if (!rest) {
throw Exception(
Expand Down
14 changes: 8 additions & 6 deletions be/src/storage/index/snii/writer/snii_compound_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <algorithm>
#include <utility>

#include "cloud/config.h"
#include "common/config.h"
#include "storage/index/snii/common/slice.h"
#include "storage/index/snii/encoding/byte_sink.h"
Expand Down Expand Up @@ -662,13 +663,14 @@ Status SniiCompoundWriter::write_tail() {
// while the filler bytes stay on disk: strictly worse than never having padded, until
// compaction rewrites the container.
//
// Gated on enable_file_cache because the saving is realised only by CachedRemoteFileReader.
// That flag defaults to FALSE; without this check a storage-compute-coupled or local-filesystem
// deployment appends up to a block of zeros per container and never reads through a block cache
// at all. (exec_env_init only validates file_cache_each_block_size when the cache is on, so in
// that configuration the value here would also be entirely unvalidated.)
// Gated on cloud mode because the saving is realised only by CachedRemoteFileReader, and only
// in cloud mode is every container read through it. A storage-compute-coupled BE writes its
// containers to local disk and reads them through LocalFileReader whether enable_file_cache is
// on (the default) or not, so padding there would append up to a block of zeros per container
// that no block cache ever repays. Cloud mode always runs with the file cache on, so
// exec_env_init has validated file_cache_each_block_size before it is read here.
const int64_t block = config::file_cache_each_block_size;
if (config::enable_file_cache && block > 0) {
if (config::is_cloud_mode() && block > 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve tail alignment for rowsets that will be cooled to remote storage. A coupled tablet with a storage policy writes its SNII index locally, then BetaRowset::upload_to() copies that file unchanged to remote storage. Its later IndexFileReader uses CachedRemoteFileReader, whose EOF read back-pads a partial final block by one whole cache block. This cloud-only gate suppresses padding for those eligible local writes, losing the previous cold-read saving. Account for cooldown when choosing the gate or retain alignment for remotely cached rowsets.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a deliberate trade-off, and we keep the cloud-mode gate:

  • Most SNII containers on a storage-compute-coupled BE are never cooled down. Padding all of them, which is what branch-4.2 does since it gates on enable_file_cache (now true by default), leaves up to half a cache block of zeros per container on local disk until compaction rewrites it.
  • Without padding, a container of 32 or more blocks that does cool down costs one extra cache block (1 MiB by default) on its first cold tail read; after that the block is cached.
  • Gating on the tablet's storage policy would mean plumbing that information through IndexFileWriter into SniiCompoundWriter, and it would still miss policies added after the data was written.

A cooldown-aware gate can be a follow-up if the cold-read cost shows up in practice.

const uint64_t unpadded = out_->bytes_written() + tail_sink.buffer().size();
const auto block_size = static_cast<uint64_t>(block);
const uint64_t pad = (block_size - unpadded % block_size) % block_size;
Expand Down
6 changes: 3 additions & 3 deletions be/src/storage/index/snii/writer/snii_compound_writer.h
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ class SniiRewriteSnapshot;
// for each logical index, in add order:
// [Core metadata][SampledTermIndex blob][DICT block directory blob]
// [metadata directory] raw SniiMetadataDirectoryPB bytes
// [block padding] OPTIONAL run of zero bytes, see write_tail(). Written only for
// containers of >= kMinPaddingLeverage cache blocks, and only when the
// file cache is on, so that the container ends on a block boundary.
// [block padding] OPTIONAL run of zero bytes, see write_tail(). Written only in cloud
// mode and only for containers of >= kMinPaddingLeverage cache blocks,
// so that the container ends on a block boundary.
// Referenced by nothing and read by nobody -- but it means the metadata
// directory is NOT necessarily adjacent to the tail pointer, and that
// container length is not a pure function of the indexed content (two
Expand Down
44 changes: 44 additions & 0 deletions be/test/io/cache/block_file_cache_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4260,6 +4260,50 @@ TEST_F(BlockFileCacheTest, create_file_caches_preserves_config_order) {
}
}

TEST_F(BlockFileCacheTest, create_file_caches_rejects_a_factory_left_without_any_cache) {
reset_file_cache_factory_for_test();
// A regular file where a cache directory would go: no cache can be created below it.
const std::string blocker = (caches_dir / "cache_path_blocker").string();
const std::string broken_path = (caches_dir / "cache_path_blocker" / "cache" / "").string();
const std::string usable_path = (caches_dir / "usable_cache" / "").string();
fs::remove_all(blocker);
fs::remove_all(usable_path);
std::ofstream(blocker).close();
Defer cleanup {[&] {
reset_file_cache_factory_for_test();
fs::remove_all(blocker);
fs::remove_all(usable_path);
}};
auto cache_path = [](const std::string& path) {
return CachePath(path, 90, 30, DEFAULT_NORMAL_PERCENT, DEFAULT_DISPOSABLE_PERCENT,
DEFAULT_INDEX_PERCENT, DEFAULT_TTL_PERCENT, "disk");
};
std::vector<std::string> skipped;
auto skip_broken = [&skipped](const std::string& path, const Status&) {
skipped.push_back(path);
return true;
};

// ignore_broken_disk skips the only path. get_by_path() would then divide by zero, so the
// factory must report failure rather than an empty success.
EXPECT_FALSE(FileCacheFactory::instance()
->create_file_caches({cache_path(broken_path)}, skip_broken)
.ok());
EXPECT_EQ(skipped, std::vector<std::string> {broken_path});
EXPECT_EQ(FileCacheFactory::instance()->get_cache_instance_size(), 0);

// With a usable path left, the broken one is still skipped and the usable one is kept.
skipped.clear();
ASSERT_TRUE(FileCacheFactory::instance()
->create_file_caches({cache_path(broken_path), cache_path(usable_path)},
skip_broken)
.ok());
EXPECT_EQ(skipped, std::vector<std::string> {broken_path});
ASSERT_EQ(FileCacheFactory::instance()->get_cache_instance_size(), 1);
EXPECT_EQ(FileCacheFactory::instance()->get_caches()[0]->get_base_path(), usable_path);
wait_until_cache_ready(*FileCacheFactory::instance()->get_caches()[0]);
}

TEST_F(BlockFileCacheTest, test_factory_2) {
if (fs::exists(cache_base_path)) {
fs::remove_all(cache_base_path);
Expand Down
3 changes: 3 additions & 0 deletions be/test/io/fs/hdfs_file_system_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ TEST(HdfsFileSystemTest, RejectsCacheBlockSizeNotDividingHdfsBatch) {
EXPECT_TRUE(io::validate_hdfs_write_batch_buffer_size(1, 256 * 1024).ok());
EXPECT_FALSE(io::validate_hdfs_write_batch_buffer_size(1, 640 * 1024).ok());
EXPECT_FALSE(io::validate_hdfs_write_batch_buffer_size(1, 2 * 1024 * 1024).ok());
// ExecEnv::init_file_cache_factory relies on this rejection before it divides by the size.
EXPECT_FALSE(io::validate_hdfs_write_batch_buffer_size(1, 0).ok());
EXPECT_FALSE(io::validate_hdfs_write_batch_buffer_size(1, -1).ok());
}

TEST(HdfsFileSystemTest, Write) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
#include <utility>
#include <vector>

#include "cloud/config.h"
#include "common/config.h"
#include "common/status.h"
#include "storage/index/snii/common/slice.h"
Expand Down Expand Up @@ -396,16 +397,20 @@ Status begin_scoring_session_from_local_input(SniiCompoundWriter* compound,
// compaction and load produce different images for the same input.
TEST(SniiStreamedWriterSessionTest, StreamedContainerPadsIdenticallyToOrdinary) {
const int64_t saved_block = doris::config::file_cache_each_block_size;
const bool saved_cache = doris::config::enable_file_cache;
const std::string saved_deploy_mode = doris::config::deploy_mode;
const std::string saved_cloud_unique_id = doris::config::cloud_unique_id;
doris::Defer restore {[&] {
doris::config::file_cache_each_block_size = saved_block;
doris::config::enable_file_cache = saved_cache;
doris::config::deploy_mode = saved_deploy_mode;
doris::config::cloud_unique_id = saved_cloud_unique_id;
}};
doris::config::cloud_unique_id.clear();

// Measure the unpadded size first, then pick a block the gate must accept: the container
// spans at least kMinPaddingLeverage blocks (we start the search at 2x that leverage and walk
// the block size up), and pad is asserted to be both due and worth paying.
doris::config::enable_file_cache = false;
// Measure the unpadded size first, outside cloud mode where nothing is padded, then pick a
// block the gate must accept: the container spans at least kMinPaddingLeverage blocks (we start
// the search at 2x that leverage and walk the block size up), and pad is asserted to be both due
// and worth paying.
doris::config::deploy_mode = "";
MemoryFile unpadded_file;
assert_ok(write_streamed_index(representative_input(), &unpadded_file));
const size_t unpadded = unpadded_file.data().size();
Expand All @@ -425,7 +430,7 @@ TEST(SniiStreamedWriterSessionTest, StreamedContainerPadsIdenticallyToOrdinary)
ASSERT_LT(2 * pad, block_size) << "the gate would decline this padding as not worth its cost";
const auto block = static_cast<int64_t>(block_size);

doris::config::enable_file_cache = true;
doris::config::deploy_mode = "cloud";
doris::config::file_cache_each_block_size = block;
MemoryFile ordinary;
MemoryFile streamed;
Expand Down
44 changes: 30 additions & 14 deletions be/test/storage/index/snii/writer/snii_compound_writer_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
#include <utility>
#include <vector>

#include "cloud/config.h"
#include "common/config.h"
#include "common/status.h"
#include "gen_cpp/snii.pb.h"
Expand Down Expand Up @@ -1491,16 +1492,24 @@ TEST(SniiCompoundWriter, PacksEachIndexSectionsContiguouslyWithoutInterleaving)
namespace {

// Writes the standard two-index fixture at the given cache block size and returns the bytes.
// The file cache is forced on by default: padding is gated on it and it defaults to OFF, so a
// fixture that left it alone would exercise only the not-padded branch whatever block size it picks.
std::vector<uint8_t> WriteFixtureAtBlockSize(int64_t block_size, bool file_cache_on = true) {
// Cloud mode is forced on by default: padding is gated on it and a UT process is not in cloud
// mode, so a fixture that left it alone would exercise only the not-padded branch whatever block
// size it picks. The file cache is on either way, as it is by default in both deploy modes, so the
// deploy mode is the only thing that decides.
std::vector<uint8_t> WriteFixtureAtBlockSize(int64_t block_size, bool cloud_mode = true) {
const int64_t saved_block = doris::config::file_cache_each_block_size;
const bool saved_cache = doris::config::enable_file_cache;
const std::string saved_deploy_mode = doris::config::deploy_mode;
const std::string saved_cloud_unique_id = doris::config::cloud_unique_id;
doris::config::file_cache_each_block_size = block_size;
doris::config::enable_file_cache = file_cache_on;
doris::config::enable_file_cache = true;
doris::config::deploy_mode = cloud_mode ? "cloud" : "";
doris::config::cloud_unique_id.clear();
doris::Defer restore {[&] {
doris::config::file_cache_each_block_size = saved_block;
doris::config::enable_file_cache = saved_cache;
doris::config::deploy_mode = saved_deploy_mode;
doris::config::cloud_unique_id = saved_cloud_unique_id;
}};

const std::string path = TempPath();
Expand Down Expand Up @@ -1614,28 +1623,35 @@ TEST(SniiCompoundWriter, SkipsPaddingWhenItWouldBeLargeRelativeToContainer) {
ExpectFixtureReadable(file);
}

// Padding is repaid only by CachedRemoteFileReader's block alignment. enable_file_cache defaults
// to OFF, and with it off there is no block cache in the read path at all -- so a container that
// qualifies on every other count must still come out unpadded.
TEST(SniiCompoundWriter, SkipsPaddingWhenTheFileCacheIsOff) {
// Padding is repaid only by CachedRemoteFileReader's block alignment, and only cloud mode reads
// containers through it. A storage-compute-coupled BE reads them from local disk even with the
// file cache on, which it is by default -- so outside cloud mode a container that qualifies on
// every other count must still come out unpadded.
TEST(SniiCompoundWriter, SkipsPaddingOutsideCloudMode) {
const size_t unpadded = WriteFixtureAtBlockSize(0).size();
const auto block = CheapPaddingBlockSize(unpadded);
ASSERT_GE(block, 2) << "fixture too small to derive a usable block size";
// Same block size the acceptance test uses, so the cache flag is the only difference.
ASSERT_NE(WriteFixtureAtBlockSize(block, /*file_cache_on=*/true).size(), unpadded)
<< "precondition: this block size must pad when the cache is on";
// Same block size the acceptance test uses, so the deploy mode is the only difference.
ASSERT_NE(WriteFixtureAtBlockSize(block, /*cloud_mode=*/true).size(), unpadded)
<< "precondition: this block size must pad in cloud mode";

const std::vector<uint8_t> file = WriteFixtureAtBlockSize(block, /*file_cache_on=*/false);
EXPECT_EQ(file.size(), unpadded) << "padded despite there being no block cache to repay it";
const std::vector<uint8_t> file = WriteFixtureAtBlockSize(block, /*cloud_mode=*/false);
EXPECT_EQ(file.size(), unpadded) << "padded outside cloud mode, where no block cache repays it";
ExpectFixtureReadable(file);
}

// Guards the arithmetic in the other direction: with the block size disabled the writer must not
// consult it at all, rather than treating 0 as "align to everything".
TEST(SniiCompoundWriter, AddsNoPaddingWhenBlockSizeIsDisabled) {
const int64_t saved_block = doris::config::file_cache_each_block_size;
const std::string saved_deploy_mode = doris::config::deploy_mode;
doris::config::file_cache_each_block_size = 0;
doris::Defer restore {[&] { doris::config::file_cache_each_block_size = saved_block; }};
// Cloud mode, so that the disabled block size is the only thing that can turn padding off.
doris::config::deploy_mode = "cloud";
doris::Defer restore {[&] {
doris::config::file_cache_each_block_size = saved_block;
doris::config::deploy_mode = saved_deploy_mode;
}};

const std::string path = TempPath();
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2363,7 +2363,7 @@ public boolean isEnableHboNonStrictMatchingMode() {
@VarAttrDef.VarAttr(name = ENABLE_FILE_CACHE, needForward = true, description = "Set wether to use file cache. "
+ "This variable takes effect only if the BE config enable_file_cache=true. "
+ "The cache is not used when BE config enable_file_cache=false.")
public boolean enableFileCache = false;
public boolean enableFileCache = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Avoid caching mutable TVF files without a versioned key. HTTP and generic file TVFs create TBrokerFileStatus without modificationTime, so their ranges reach BE with mtime 0. This new session default makes FileFactory cache them, while CachedRemoteFileReader hashes only path:mtime. If a stable HTTP URL or object key is overwritten with different same-size rows, the next query reuses path:0 blocks and returns stale rows. Supply a reliable version for the cache key or keep caching off for these unversioned ranges.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed for the generic file TVFs, fixed in 0911a80. Since #62023 moved TVF listing onto the filesystem SPI, parseFile dropped FileEntry#modificationTime when building TBrokerFileStatus, so s3() / hdfs() ranges reached BE with mtime 0. They now carry the listed modification time again, as on branch-4.x (BrokerUtil.parseFile copies it). Covered by ExternalFileTableValuedFunctionTest#testListedFilesCarryTheirModificationTime.

http() has never carried a version on any branch, including 4.1.5 / 4.2 where the cache is already on by default. Making it safe needs a validator from the response (Last-Modified / ETag), or bypassing the cache when there is none. That is a change to the HTTP TVF itself, so it will be tracked separately rather than in this default-value PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Namespace external cache entries by the actual source. Two S3 TVFs can read the same s3://bucket/key from different s3.endpoint values, or two HTTP TVFs can request different equal-length representations of one URL using different headers. Their TVF mtime is 0, and the new default sends both through a cache keyed only by path:mtime; the second query on the same BE can return the first source's rows even when both files are immutable. Include a stable endpoint/request-representation identity in the key or bypass shared caching for these sources.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache key layout (path:mtime, with no endpoint or request identity) is the existing design for all external reads, on master and branch-4.x alike; this PR only changes the defaults. The main exposure described here, TVF ranges with mtime 0, is removed by 0911a80: two S3 / HDFS sources now collide only if they share both the path and the modification time. Adding the source identity (endpoint / filesystem identity, and the HTTP request representation) to the external cache key belongs in the file cache layer and invalidates existing cache entries, so it will be tracked as a separate change.


// Specify base path for file cache, or chose a random path.
@VarAttrDef.VarAttr(name = FILE_CACHE_BASE_PATH, needForward = true, description = "Specify the storage path of "
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,8 +184,12 @@ protected void parseFile() throws AnalysisException {
entries = fs.listFiles(Location.of(path));
}
for (FileEntry e : entries) {
fileStatuses.add(new TBrokerFileStatus(
e.location().uri(), e.isDirectory(), e.length(), !e.isDirectory()));
TBrokerFileStatus status = new TBrokerFileStatus(
e.location().uri(), e.isDirectory(), e.length(), !e.isDirectory());
// BE keys cached blocks of external files by path and modification time, so a
// file overwritten in place must arrive with its new mtime, not as mtime 0.
status.setModificationTime(e.modificationTime());
fileStatuses.add(status);
}
} catch (IOException e) {
throw new UserException("list files failed for path " + path + ": " + e.getMessage(), e);
Expand Down
Loading
Loading