Skip to content

[python] Add list-files & file command support for Python CLI. - #11154

Open
fengruotj wants to merge 2 commits into
apache:masterfrom
fengruotj:feat/pypaimon-cli-file
Open

fengruotj wants to merge 2 commits into
apache:masterfrom
fengruotj:feat/pypaimon-cli-file

Conversation

@fengruotj

Copy link
Copy Markdown
Contributor

Purpose

Implements the file commands proposed in #11073 (https://github.com/apache/paimon/issues/11073).

Adds read-only file inspection to the PyPaimon CLI:

  • paimon table list-files <database.table> lists live data files referenced by the latest or a specified snapshot, including the snapshot ID and file metadata. It supports --snapshot-id, --bucket, and --limit, with table output by default or JSON output.
  • paimon table file <database.table> --file reads a Parquet file at a path or filesystem URI and displays its schema and footer metadata, including row groups and column statistics. It supports JSON output by default or table output, with schema and metadata displayed as indented, multiline JSON.

These commands inspect metadata without scanning data records or modifying the table. The file command does not query snapshots or manifests. CLI documentation and output examples are included.

Also renames table rollback --snapshot to --snapshot-id for consistency with the other table commands.

Tests

  • paimon-python/pypaimon/tests/cli_table_test.py: 90 passed, including file inspection regression tests.
  • flake8 for the changed Python files: passed.

@fengruotj fengruotj changed the title [python] Add list-files & file by file support for Python CLI. [python] Add list-files & file command support for Python CLI. Nov 14, 2026
table = _open_schema_table(args)
try:
status = table.file_io.get_file_status(args.file)
if status.type != pafs.FileType.File:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Med]
list-files and manifest emit _metadata_file_path(table, entry), which preserves file: schemes and percent-escaped partition directories (e.g., file:/warehouse/db.db/t/pt=a%2Fb/bucket-0/data-0.parquet). Passing args.file directly to table.file_io causes LocalFileIO._to_file() to URL-decode %2F into / and fail with FileNotFoundError.

We should convert the path with to_file_io_path(args.file) before calling file_io (matching DataFileMeta.physical_path()):

        from pypaimon.utils.path import to_file_io_path

        io_path = to_file_io_path(args.file)
        status = table.file_io.get_file_status(io_path)
        file_format = _data_file_format(basename(status.path)).lower()
        if file_format != 'parquet':
            raise ValueError(
                f"Unsupported data file format '{file_format}' for '{args.file}'; only parquet is supported.")

        with table.file_io.new_input_stream(io_path) as stream:

@fengruotj fengruotj Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Akash3121 Thanks for reviewing! Agreed. I've added a local change to preserve the emitted metadata path at both FileIO calls, with regression tests for the public list-files/manifest-to-file round trip on escaped partitions and for ordinary encoded file URIs.

Comment thread paimon-python/pypaimon/cli/cli_table.py Outdated
rollback_target = rollback_parser.add_mutually_exclusive_group(required=True)
rollback_target.add_argument(
'--snapshot', type=int, default=None,
'--snapshot-id', type=int, default=None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unifying this option name with --snapshot-id used by get, snapshot, list-manifests, and list-files makes the CLI much more consistent, and argparse sets args.snapshot_id automatically.

Two minor suggestions here:

  1. Keep --snapshot as a backward-compatible alias ('--snapshot-id', '--snapshot') so existing scripts using paimon table rollback ... --snapshot <id> don't break.
  2. Could this also use type=_positive_int like the --snapshot-id option on get (line 1471), snapshot (line 1501), list-manifests (line 1531), and list-files (line 1564)?
Suggested change
'--snapshot-id', type=int, default=None,
'--snapshot-id', '--snapshot', type=int, default=None,

@fengruotj fengruotj Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Akash3121 Thanks for reviewing! I've added --snapshot as an explicit alias and switched to _positive_int.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

I independently confirmed the existing path round-trip finding on bdd9b2a142 with a real filesystem catalog and committed Parquet data: warehouse configured as a file URI, partition value a/b, then public list-files --format json followed by file --file <exact emitted file_path>. The latter exits 1 because the literal pt=a%2Fb directory is URL-decoded; inspecting the same file using its physical local path succeeds. This breaks the documented diagnostic workflow. Please preserve the metadata path semantics at both FileIO calls and add this public command-to-command regression, while retaining support for ordinary encoded filesystem URIs.

90 CLI tests passed (108 subtests); project flake8 and git diff --check passed. The explicit legacy rollback --snapshot 1 command also succeeded: argparse currently accepts it as an unambiguous abbreviation, so I did not find a compatibility failure there. No additional independent code finding.

@fengruotj

fengruotj commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

@JingsongLi Thanks for reviewing! Agreed. I’ve updated both FileIO calls to preserve the metadata path semantics, with regression tests for the public list-files → file round trip on escaped partitions and for ordinary encoded filesystem URIs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants