Skip to content

Redesign the access to meta version - #436

Merged
morningman merged 8 commits into
apache:masterfrom
morningman:meta_version
Dec 17, 2018
Merged

morningman merged 8 commits into
apache:masterfrom
morningman:meta_version

Conversation

@morningman

@morningman morningman commented Dec 16, 2018 •

Copy link
Copy Markdown
Contributor

Because the meta version is only be used in catalog saving and loading.
So currently this version is a field of Catalog class. And we can get this
version only by calling Catalog.getCurrentCatalogJournalVersion().

But in restore process, we need to read the meta data which is saved with
a specified meta version. So we need a flexible way to read a meta data
using a specified meta version, not only the version from Catalog.

So we create a new class called MetaContext. Currently it only has one field,
'journalVersion', to save the current journal version. And it is a
thread local variable, so that we can create a MetaContext anywhere we want,
and setting the 'journalVersion' which we want to use for reading meta.

Currently, there are 4 threads which is related to meta data saving and loading.

  1. The Frontend starting thread, which will call Catalog.initialize() to load the image.
  2. the Frontend state listener thread, which will listen the state changing, and call
    transferToMaster() or transferToNonMaster().
  3. Edit log replayed thread, which is created when calling transferToNonMaster().
    It will replay edit log
  4. Checkpoint thread, which is created when calling transferToMaster(). It will do
    the checkpoint periodically.

Notice that we get the 'current meta version' only when 'READING' the meta (not WRITING).
So we only need to take care of all 'READING' threads.
We create MetaContext thread local variable for these 4 threads, and thread 2,3,4's
meta context inherit from thread 1's meta context. Because thread 1 will load the origin
image file and get the very first meta version.

And we leave the Catalog.getCurrentCatalogJournalVersion()'s name unchanged, just
change its content, because we don't want change a lot codes this time.

On the other hand, we add the current meta version in backup job info file when doing
backup job. So that when restoring from a backup snapshot, we can know which meta
version we should use for read the meta.
And also , we add a new property "meta_version" for Restore stmt, so that we can specify
the meta version used for reading backup meta. It is for those old backup snapshots
which do not has meta version saving in backup job info file.

meta version(journal version) is currently a field of Catalog.
Each time we want to judge the meta version when we read the journal,
we call Catalog.getCurrentCatalogJournalVersion().

This is no flexible to get the meta version anywhere.
Currently, meta version can only be set and get in main Catalog thread
and checkpoint catalog thread. But for some cases, we want to decide
our own meta version when reading a journal, not the version got from
Catalog.

So we create a new class called MetaContext. Currently it only has a field
'journalVersion' to save the current journal version. And it has is a
thread local variable, so that we can create a MetaContext anywhere we want,
and setting the 'journalVersion' which we want to use for reading journal.
@morningman morningman changed the title Redesign meta version Redesign the access to meta version Dec 16, 2018
@morningman
morningman merged commit 45e42bd into apache:master Dec 17, 2018
morningman added a commit to morningman/doris that referenced this pull request Nov 18, 2026
### Background

ORC physical stripe readers reuse parsed immutable file metadata and lazy row prefixes. The SDK implementation has merged into the upstream orc branch.

### What problem does this PR solve?

Issue Number: None
Related PR: apache/doris-thirdparty#436 (merged SDK), apache#70664 (consumer)
PR Type: Release/Dependency

Problem Summary:

Replace the local SDK pin with its upstream squash-merge commit so the Doris dependency is available from the maintained branch.

### Details

- **Versions:** contrib/apache-orc 89c98c2 to upstream 00fa6d6.
- **Upstream changes:** Both commits have identical complete source trees; apache#436 combines parsed file-tail sharing with lazy prefix initialization.
- **Compatibility:** Source and test contents are unchanged. Native consumers build against the extended SDK interface already used by this branch.
- **Rollback:** Restore the prior pin if the dependency update must be reverted.

### Testing

Verified identical SDK tree hashes. The prior local ASAN build and 255 selected tests cover these exact SDK source bytes; they were not rerun for the hash-only pin update.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Codex
- Generated: Dependency update and commit description
- Human review: not stated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants