Repository navigation
[improvement](filecache) Enable file cache by default #68797
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
19aaf47
2a7d02f
368abfc
57874cb
e3d1f32
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1228,7 +1228,7 @@ DEFINE_Validator(variant_storage_parse_mode, | |
| [](const int config) -> bool { return config >= 0 && config <= 2; }); | ||
|
|
||
| // block file cache | ||
| DEFINE_Bool(enable_file_cache, "false"); | ||
| DEFINE_Bool(enable_file_cache, "true"); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Validate the cache block size before enabling it by default. A coupled BE that previously ran with
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 0b2a2f2. |
||
| // ATTENTION: For test only. Keep this enabled in production. | ||
| // Whether S3 storage write paths populate file cache while writing data to object storage. | ||
| // Disable this for tests that need load and compaction output to bypass file cache while keeping | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ | |
| #include <algorithm> | ||
| #include <utility> | ||
|
|
||
| #include "cloud/config.h" | ||
| #include "common/config.h" | ||
| #include "storage/index/snii/common/slice.h" | ||
| #include "storage/index/snii/encoding/byte_sink.h" | ||
|
|
@@ -662,13 +663,14 @@ Status SniiCompoundWriter::write_tail() { | |
| // while the filler bytes stay on disk: strictly worse than never having padded, until | ||
| // compaction rewrites the container. | ||
| // | ||
| // Gated on enable_file_cache because the saving is realised only by CachedRemoteFileReader. | ||
| // That flag defaults to FALSE; without this check a storage-compute-coupled or local-filesystem | ||
| // deployment appends up to a block of zeros per container and never reads through a block cache | ||
| // at all. (exec_env_init only validates file_cache_each_block_size when the cache is on, so in | ||
| // that configuration the value here would also be entirely unvalidated.) | ||
| // Gated on cloud mode because the saving is realised only by CachedRemoteFileReader, and only | ||
| // in cloud mode is every container read through it. A storage-compute-coupled BE writes its | ||
| // containers to local disk and reads them through LocalFileReader whether enable_file_cache is | ||
| // on (the default) or not, so padding there would append up to a block of zeros per container | ||
| // that no block cache ever repays. Cloud mode always runs with the file cache on, so | ||
| // exec_env_init has validated file_cache_each_block_size before it is read here. | ||
| const int64_t block = config::file_cache_each_block_size; | ||
| if (config::enable_file_cache && block > 0) { | ||
| if (config::is_cloud_mode() && block > 0) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P2] Preserve tail alignment for rowsets that will be cooled to remote storage. A coupled tablet with a storage policy writes its SNII index locally, then
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a deliberate trade-off, and we keep the cloud-mode gate:
A cooldown-aware gate can be a follow-up if the cold-read cost shows up in practice. |
||
| const uint64_t unpadded = out_->bytes_written() + tail_sink.buffer().size(); | ||
| const auto block_size = static_cast<uint64_t>(block); | ||
| const uint64_t pad = (block_size - unpadded % block_size) % block_size; | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2363,7 +2363,7 @@ public boolean isEnableHboNonStrictMatchingMode() { | |
| @VarAttrDef.VarAttr(name = ENABLE_FILE_CACHE, needForward = true, description = "Set wether to use file cache. " | ||
| + "This variable takes effect only if the BE config enable_file_cache=true. " | ||
| + "The cache is not used when BE config enable_file_cache=false.") | ||
| public boolean enableFileCache = false; | ||
| public boolean enableFileCache = true; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Avoid caching mutable TVF files without a versioned key. HTTP and generic file TVFs create
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed for the generic file TVFs, fixed in 0911a80. Since #62023 moved TVF listing onto the filesystem SPI,
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [P1] Namespace external cache entries by the actual source. Two S3 TVFs can read the same
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The cache key layout ( |
||
|
|
||
| // Specify base path for file cache, or chose a random path. | ||
| @VarAttrDef.VarAttr(name = FILE_CACHE_BASE_PATH, needForward = true, description = "Specify the storage path of " | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[P1] Ensure the default cache has a usable directory before enabling it. A coupled BE can have writable
storage_root_path=/data/dorisbut read-only${DORIS_HOME}; the default cache path${DORIS_HOME}/file_cachethen fails creation and this new true default makes BE startup fail. Withignore_broken_disk=true, the failed sole cache is skipped and startup succeeds with_cachesempty; the first cached read orRowset::clear_cache()callsget_by_path(hash), which takes% _caches.size()and crashes. Please provide a usable default/fallback and reject or disable caching when every path is skipped.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, two parts:
create_file_cachesnow fails when every configured path was skipped, so startup stops with "failed to init file cache ... All N file cache paths are broken", the same waydoris_mainhandles storage and spill paths ("All disks are broken, exit."). The crash would in fact come before the first cached read:Rowset::clear_cache()looks up the cache for every removed rowset whileenable_file_cacheis on. Covered by the newBlockFileCacheTest.create_file_caches_rejects_a_factory_left_without_any_cache.${DORIS_HOME}/file_cachecannot be created, and how to setfile_cache_pathorenable_file_cache = false.