Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 0 additions & 18 deletions vortex-array/src/aggregate_fn/plugin.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ use crate::aggregate_fn::AggregateFn;
use crate::aggregate_fn::AggregateFnId;
use crate::aggregate_fn::AggregateFnRef;
use crate::aggregate_fn::AggregateFnVTable;
use crate::dtype::DType;

/// Reference-counted pointer to an aggregate function plugin.
pub type AggregateFnPluginRef = Arc<dyn AggregateFnPlugin>;
Expand All @@ -29,19 +28,6 @@ pub trait AggregateFnPlugin: 'static + Send + Sync {
/// Deserialize an aggregate function from serialized metadata.
fn deserialize(&self, metadata: &[u8], session: &VortexSession)
-> VortexResult<AggregateFnRef>;

/// The default zone statistic (per-chunk) for a column of `input_dtype`, or `None` if the
/// dtype is not supported (or this aggregate is not a zone statistic at all).
///
/// This is how a registered aggregate volunteers itself as a per-chunk statistic: when a
/// zoned layout writer opens a column, it collects every plugin's default via
/// [`AggregateFnSession::zone_stat_defaults`], so new statistics can be added without the
/// writer knowing about them.
///
/// [`AggregateFnSession::zone_stat_defaults`]: crate::aggregate_fn::session::AggregateFnSession::zone_stat_defaults
fn zone_stat_default(&self, _input_dtype: &DType) -> Option<AggregateFnRef> {
None
}
}

impl std::fmt::Debug for dyn AggregateFnPlugin {
Expand All @@ -65,8 +51,4 @@ impl<V: AggregateFnVTable> AggregateFnPlugin for V {
let options = AggregateFnVTable::deserialize(self, metadata, session)?;
Ok(AggregateFn::new(self.clone(), options).erased())
}

fn zone_stat_default(&self, input_dtype: &DType) -> Option<AggregateFnRef> {
AggregateFnVTable::zone_stat_default(self, input_dtype)
}
}
18 changes: 0 additions & 18 deletions vortex-array/src/aggregate_fn/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ use vortex_utils::aliases::hash_map::HashMap;

use crate::aggregate_fn::AggregateFnId;
use crate::aggregate_fn::AggregateFnPluginRef;
use crate::aggregate_fn::AggregateFnRef;
use crate::aggregate_fn::AggregateFnVTable;
use crate::aggregate_fn::ExtensionStoragePushdown;
use crate::aggregate_fn::fns::all_nan::AllNan;
Expand Down Expand Up @@ -57,7 +56,6 @@ use crate::arrays::runend::compute::is_sorted::RunEndIsSortedKernel;
use crate::arrays::runend::compute::min_max::RunEndMinMaxKernel;
use crate::arrays::runend::compute::sum::RunEndSumKernel;
use crate::arrays::widen::compute::aggregate::WidenArrayAggregate;
use crate::dtype::DType;
use crate::dtype::extension::ExtVTable;
use crate::extension::datetime::Date;
use crate::extension::datetime::Time;
Expand Down Expand Up @@ -197,22 +195,6 @@ impl AggregateFnSession {
self.register(ExtensionStoragePushdown::new(MinMax, ext));
}

/// The default per-chunk zone statistics for a column of `input_dtype`, collected from every
/// registered aggregate's `zone_stat_default`.
///
/// Each call scans the whole plugin registry, so this is intended to be called once per
/// column when a zoned writer is opened, not per chunk or per row.
pub fn zone_stat_defaults(&self, input_dtype: &DType) -> Vec<AggregateFnRef> {
self.registry.read(|registry| {
let mut fns: Vec<AggregateFnRef> = registry
.values()
.filter_map(|plugin| plugin.zone_stat_default(input_dtype))
.collect();
fns.sort_by_key(|f| f.id());
fns
})
}

/// Returns the aggregate kernel registered for `array_id` and `agg_fn_id`, if any.
///
/// Lookup first checks for a kernel registered for the exact aggregate function, then falls
Expand Down
6 changes: 0 additions & 6 deletions vortex-array/src/aggregate_fn/vtable.rs
Original file line number Diff line number Diff line change
Expand Up @@ -169,12 +169,6 @@ pub trait AggregateFnVTable: 'static + Sized + Clone + Send + Sync {
/// Returns `None` if the aggregate function cannot be applied to the input dtype.
fn return_dtype(&self, options: &Self::Options, input_dtype: &DType) -> Option<DType>;

/// If this aggregate should be computed as a default zone statistic for `input_dtype`, return
/// the bound aggregate to store. Default: not a zone-map default.
fn zone_stat_default(&self, _input_dtype: &DType) -> Option<AggregateFnRef> {
None
}

/// DType of the intermediate partial accumulator state.
///
/// Use a struct dtype when multiple fields are needed
Expand Down
33 changes: 9 additions & 24 deletions vortex-layout/src/layouts/zoned/writer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ use vortex_array::aggregate_fn::fns::max::Max;
use vortex_array::aggregate_fn::fns::min::Min;
use vortex_array::aggregate_fn::fns::nan_count::NanCount;
use vortex_array::aggregate_fn::fns::null_count::NullCount;
use vortex_array::aggregate_fn::session::AggregateFnSessionExt;
use vortex_array::dtype::DType;
use vortex_error::VortexError;
use vortex_error::VortexResult;
Expand Down Expand Up @@ -114,7 +113,7 @@ impl LayoutStrategy for ZonedStrategy {
.options
.aggregate_fns
.clone()
.unwrap_or_else(|| default_zoned_aggregate_fns(stream.dtype(), session));
.unwrap_or_else(|| default_zoned_aggregate_fns(stream.dtype()));
// Fields of a nullable struct are written unmasked, so every aggregate over them sees
// values the logical column does not contain: counts come out wrong and min/max come out
// wider than the truth. Zone maps hold exact values only, so until the aggregates are
Expand Down Expand Up @@ -313,7 +312,7 @@ impl ZonedStrategy {
}
}

fn default_zoned_aggregate_fns(dtype: &DType, session: &VortexSession) -> Arc<[AggregateFnRef]> {
fn default_zoned_aggregate_fns(dtype: &DType) -> Arc<[AggregateFnRef]> {
let (max, min) = match dtype {
DType::Utf8(_) | DType::Binary(_) => (
BoundedMax.bind(BoundedMaxOptions {
Expand All @@ -330,27 +329,17 @@ fn default_zoned_aggregate_fns(dtype: &DType, session: &VortexSession) -> Arc<[A
};

// Sum is deliberately absent: zone maps exist to prune, and a zone sum prunes nothing.
// Its semantics are also unsettled - null-on-empty was changed in #9113 and reverted in
// #9324 - so it is not a stat to record in every zone of every file, let alone freeze
// into an edition. File-level statistics still record a sum by default (see
// `file_stats::default_pruning_aggregate_fns`).
let mut aggregate_fns = vec![
vec![
max,
min,
NanCount.bind(EmptyOptions),
NullCount.bind(EmptyOptions),
];

// Stats from spatial extension types are discovered from the registry at runtime instead.
aggregate_fns.extend(session.aggregate_fns().zone_stat_defaults(dtype));

aggregate_fns.into()
]
.into()
}

#[cfg(test)]
mod tests {
use std::sync::LazyLock;

use rstest::rstest;
use vortex_array::ArrayContext;
use vortex_array::IntoArray;
Expand Down Expand Up @@ -385,18 +374,15 @@ mod tests {
use crate::sequence::SequentialArrayStreamExt;
use crate::session::LayoutSession;

static SESSION: LazyLock<VortexSession> = LazyLock::new(VortexSession::empty);

#[rstest]
#[case::variant(DType::Variant(Nullability::Nullable))]
#[case::variant_in_struct(DType::Struct(
StructFields::new(["v"].into(), vec![DType::Variant(Nullability::Nullable)]),
Nullability::NonNullable,
))]
fn zone_maps_hold_no_variant_state(#[case] dtype: DType) {
let session = VortexSession::empty().with::<LayoutSession>();
let accumulator =
AggregateStatsAccumulator::new(&dtype, &default_zoned_aggregate_fns(&dtype, &session));
AggregateStatsAccumulator::new(&dtype, &default_zoned_aggregate_fns(&dtype));
for aggregate_fn in accumulator.aggregate_fns().iter() {
let state = aggregate_state_dtype(&dtype, aggregate_fn).vortex_expect("kept state");
assert!(!contains_variant(&state), "{aggregate_fn} keeps a variant");
Expand Down Expand Up @@ -495,8 +481,7 @@ mod tests {

#[test]
fn default_aggregates_bound_variable_length_min_max() {
let aggregate_fns =
default_zoned_aggregate_fns(&DType::Utf8(Nullability::NonNullable), &SESSION);
let aggregate_fns = default_zoned_aggregate_fns(&DType::Utf8(Nullability::NonNullable));

assert_eq!(
aggregate_fns[0].as_::<BoundedMax>().max_bytes,
Expand All @@ -510,7 +495,7 @@ mod tests {

#[test]
fn default_aggregates_keep_fixed_width_min_max_exact() {
let aggregate_fns = default_zoned_aggregate_fns(&PType::I32.into(), &SESSION);
let aggregate_fns = default_zoned_aggregate_fns(&PType::I32.into());

assert!(aggregate_fns[0].is::<Max>());
assert!(aggregate_fns[1].is::<Min>());
Expand All @@ -524,7 +509,7 @@ mod tests {
Timestamp::new(TimeUnit::Microseconds, Nullability::Nullable).erased(),
))]
fn default_aggregates_never_record_sum(#[case] dtype: DType) {
let aggregate_fns = default_zoned_aggregate_fns(&dtype, &VortexSession::empty());
let aggregate_fns = default_zoned_aggregate_fns(&dtype);

assert!(
aggregate_fns
Expand Down
2 changes: 2 additions & 0 deletions vortex-spatial/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ divan = { workspace = true }
mimalloc = { workspace = true }
rstest = { workspace = true }
vortex-array = { workspace = true, features = ["_test-harness"] }
vortex-file = { workspace = true }
vortex-io = { workspace = true }
vortex-layout = { workspace = true }
vortex-spatial = { path = ".", features = ["_test-harness"] }

Expand Down
54 changes: 49 additions & 5 deletions vortex-spatial/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,61 @@ Most functions that need a general geometry algorithm use the row-function adapt

Avoid per-row `scalar_at` or validity lookups in either path. Materialize coordinates, offsets, and validity once for the batch.

## Chunk pruning
## File statistics and chunk pruning

`GeometryAabb` computes the two-dimensional axis-aligned bounding box (AABB) of a geometry column. The writer stores it as the default zone statistic for native geometry columns.
`GeometryAabb` computes the two-dimensional axis-aligned bounding box (AABB) of a geometry column. Spatial initialization registers the aggregate and pruning rules. Writers explicitly choose whether to record AABB statistics at the file level, in zone maps, or both; neither records them by default.

The current rules can prune:
To record file-wide bounds, enable nested statistics and include `GeometryAabb` in the requested aggregates:

```rust
use vortex_array::aggregate_fn::{AggregateFnVTableExt, EmptyOptions};
use vortex_array::aggregate_fn::fns::null_count::NullCount;
use vortex_file::WriteOptionsSessionExt;
use vortex_spatial::aggregate_fn::GeometryAabb;
# let session = vortex_session::VortexSession::empty();

vortex_spatial::initialize(&session);
let options = session
.write_options()
.include_nested_statistics()
.with_file_statistics(vec![
GeometryAabb.bind(EmptyOptions),
NullCount.bind(EmptyOptions),
]);
```

`with_file_statistics` replaces the default list, so include any other aggregates the application needs. File statistics accumulate the box across incoming batches and store it in the footer independently of zone maps. Consumers select the column with `FileStatistics::get_by_path` and request `GeometryAabb.bind(EmptyOptions)` from its `AggregateStats`. The value is a native `geoarrow.box`; null or empty geometries contribute no extent. The legacy file statistics format cannot store geometry bounds, so `include_nested_statistics()` is required.

For zone bounds, explicitly configure `ZonedLayoutOptions::aggregate_fns`:

```rust
use std::sync::Arc;

use vortex_array::aggregate_fn::{AggregateFnVTableExt, EmptyOptions};
use vortex_array::aggregate_fn::fns::null_count::NullCount;
use vortex_layout::layouts::zoned::writer::ZonedLayoutOptions;
use vortex_spatial::aggregate_fn::GeometryAabb;

let options = ZonedLayoutOptions {
aggregate_fns: Some(Arc::from([
GeometryAabb.bind(EmptyOptions),
NullCount.bind(EmptyOptions),
])),
..Default::default()
};
```

Pass these options to `ZonedStrategy::new`. Its input must have one chunk per zone, matching `block_size` except for the final chunk; use `RepartitionStrategy` to establish these boundaries. For a geometry field in a table, attach that strategy with `WriteStrategyBuilder::with_field_writer` and pass the built strategy to `with_strategy`. This aggregate list also replaces the zone defaults. The [statistics tests](src/tests/statistics.rs) show a complete write/read round trip and verify that explicitly requested zone bounds enable pruning.

On reads, initialize the spatial extension on the reader's session as well. Filters passed to `VortexFile::scan` automatically use the registered pruning rules and available file or zone statistics. Registering a compute function does not give it a pruning rule. The current spatial rules support:

- positive `ST_Intersects(geometry_column, constant_geometry)` filters when the boxes are disjoint;
- `<`, `<=`, `>`, and `>=` comparisons on
`ST_Distance(geometry_column, constant_geometry)` using box-distance bounds.

These proofs are conservative. If the expression has a different shape, the constant is null or empty, or the file has no AABB statistic, the chunk is kept. Box overlap alone never proves that two geometries intersect.
These rules accept a native geometry column at the scope root or through a nested field path, with the constant on either side of the spatial function. They do not infer bounds through arbitrary function calls or support spatial joins. `ST_Contains` has compute support but no pruning rule yet.

These proofs are conservative: only a proof that no row can match skips a file or zone. Missing AABB statistics supply no such proof, and box overlap alone never proves that two geometries intersect.

## Interoperability and limits

Expand All @@ -80,7 +124,7 @@ Some important limits:
- Geometry calculations use `x` and `y`. Structural operations can preserve `z` and `m`: `ST_Collect` reuses the input storage, and `ST_MakeLine` preserves and promotes point dimensions.
- There is no native `GeometryCollection` or mixed-geometry union type so far.
- WKB columns cannot be passed to the native compute functions or AABB pruning.
- AABB statistics prune chunks; they are not a spatial index or spatial-join implementation.
- AABB statistics prune files and zones; they are not a spatial index or spatial-join implementation.

When adding a serialized dtype or statistic, register it in `initialize` and add it to `src/editions.rs`.

Expand Down
Loading
Loading