Skip to content

Core: Control inferred column metrics and collect nested bounds - #4458

Open
Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:feat/2699-metrics-max-inferred-columns
Open

Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:feat/2699-metrics-max-inferred-columns

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Rationale for this change

Nested fields were downgraded to counts-only statistics in writer stats planning, so struct columns could not be pruned on bounds. That matters for schemas such as GeoParquet, which keep bounding boxes in struct fields.

Add write.metadata.metrics.max-inferred-column-defaults (default 100) and apply write.metadata.metrics.default to at most that many primitive fields, mirroring Java's MetricsConfig. With the limit in place, nested fields collect the same bounds as top-level fields.

Behavior details:

  • If write.metadata.metrics.default is set explicitly, it still applies to every column.
  • write.metadata.metrics.column.* overrides still apply to columns beyond the limit.
  • A negative limit falls back to 100.

Are these changes tested?

Yes. New tests cover the limit, overrides beyond the limit, the explicit default bypassing the limit, and the negative fallback. Existing stats tests were updated to expect nested bounds. tests/io/test_pyarrow_stats.py, tests/table, and the local write tests pass.

Are there any user-facing changes?

Yes. Nested columns now collect bounds by default, and tables with more than 100 primitive columns stop collecting inferred metrics beyond the first 100 unless they configure a default mode or per-column overrides.

Nested fields were downgraded to counts-only statistics, which prevents
bounds pruning for struct fields such as GeoParquet bounding boxes.

Add write.metadata.metrics.max-inferred-column-defaults, defaulting to
100, and apply the default mode to at most that many primitive fields,
matching the Java MetricsConfig. An explicitly configured default mode
still applies to every column, and per-column overrides still work beyond
the limit. With the limit in place, nested fields collect the same bounds
as top-level fields.
property_as_int can return None, which mypy flagged on Python 3.11.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant