Repository navigation
Redesign the access to meta version - #436
Merged
Merged
Conversation
meta version(journal version) is currently a field of Catalog. Each time we want to judge the meta version when we read the journal, we call Catalog.getCurrentCatalogJournalVersion(). This is no flexible to get the meta version anywhere. Currently, meta version can only be set and get in main Catalog thread and checkpoint catalog thread. But for some cases, we want to decide our own meta version when reading a journal, not the version got from Catalog. So we create a new class called MetaContext. Currently it only has a field 'journalVersion' to save the current journal version. And it has is a thread local variable, so that we can create a MetaContext anywhere we want, and setting the 'journalVersion' which we want to use for reading journal.
imay
approved these changes
Dec 17, 2018
morningman
added a commit
to morningman/doris
that referenced
this pull request
Nov 18, 2026
### Background ORC physical stripe readers reuse parsed immutable file metadata and lazy row prefixes. The SDK implementation has merged into the upstream orc branch. ### What problem does this PR solve? Issue Number: None Related PR: apache/doris-thirdparty#436 (merged SDK), apache#70664 (consumer) PR Type: Release/Dependency Problem Summary: Replace the local SDK pin with its upstream squash-merge commit so the Doris dependency is available from the maintained branch. ### Details - **Versions:** contrib/apache-orc 89c98c2 to upstream 00fa6d6. - **Upstream changes:** Both commits have identical complete source trees; apache#436 combines parsed file-tail sharing with lazy prefix initialization. - **Compatibility:** Source and test contents are unchanged. Native consumers build against the extended SDK interface already used by this branch. - **Rollback:** Restore the prior pin if the dependency update must be reverted. ### Testing Verified identical SDK tree hashes. The prior local ASAN build and 255 selected tests cover these exact SDK source bytes; they were not rerun for the hash-only pin update. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex - Generated: Dependency update and commit description - Human review: not stated
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Because the meta version is only be used in catalog saving and loading.
So currently this version is a field of Catalog class. And we can get this
version only by calling Catalog.getCurrentCatalogJournalVersion().
But in restore process, we need to read the meta data which is saved with
a specified meta version. So we need a flexible way to read a meta data
using a specified meta version, not only the version from Catalog.
So we create a new class called MetaContext. Currently it only has one field,
'journalVersion', to save the current journal version. And it is a
thread local variable, so that we can create a MetaContext anywhere we want,
and setting the 'journalVersion' which we want to use for reading meta.
Currently, there are 4 threads which is related to meta data saving and loading.
transferToMaster() or transferToNonMaster().
It will replay edit log
the checkpoint periodically.
Notice that we get the 'current meta version' only when 'READING' the meta (not WRITING).
So we only need to take care of all 'READING' threads.
We create MetaContext thread local variable for these 4 threads, and thread 2,3,4's
meta context inherit from thread 1's meta context. Because thread 1 will load the origin
image file and get the very first meta version.
And we leave the Catalog.getCurrentCatalogJournalVersion()'s name unchanged, just
change its content, because we don't want change a lot codes this time.
On the other hand, we add the current meta version in backup job info file when doing
backup job. So that when restoring from a backup snapshot, we can know which meta
version we should use for read the meta.
And also , we add a new property "meta_version" for Restore stmt, so that we can specify
the meta version used for reading backup meta. It is for those old backup snapshots
which do not has meta version saving in backup job info file.