Skip to content

Refactor DataFusion crate folder structure - #49

Merged
manuzhang merged 1 commit into
mainfrom
gabotechs/folder-structure
Oct 10, 2026
Merged

manuzhang merged 1 commit into
mainfrom
gabotechs/folder-structure

Conversation

@gabotechs

@gabotechs gabotechs commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Reorganizes the module structure of the project to something more granular, with the intention of improving visual indexing and better clarifying where should the different pieces be contributed.

The new module structure looks like this:

datafusion/
  src/
    catalog/ # all code related to catalog and schema handling
    metadata/ # all code related to ExecutionPlan implementations for reading metadata
    read/ # all code related to ExecutionPlan implementations for reading data
    table_provider/ # all planning-time TableProvider implementations for mapping logical to physical nodes
    write/ # all code related to ExecutionPlan implementations for writing data

/// For catalog-backed tables with write support and automatic refresh, use
/// [`IcebergTableProvider`](super::IcebergCatalogTableProvider) instead.
#[derive(Debug, Clone)]
pub struct IcebergStaticTableProvider {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Factored out from ‎crates/datafusion/src/table/mod.rs‎ untouched.

/// For catalog-backed tables with write support and automatic refresh, use
/// [`IcebergTableProvider`] instead.
#[derive(Debug, Clone)]
pub struct IcebergStaticTableProvider {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to static_table_provider.rs untouched

use tempfile::TempDir;

use super::*;

async fn get_test_table_from_metadata_file() -> Table {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to static_table_provider.rs untouched

@gabotechs
gabotechs added this pull request to stack #51 October 7, 2026 08:42

@NoahKusaba NoahKusaba left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gabotechs and @NoahKusaba

@manuzhang
manuzhang merged commit 79d68a5 into main Oct 10, 2026
4 checks passed
@manuzhang
manuzhang deleted the gabotechs/folder-structure branch October 10, 2026 04:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants