Repository navigation
Open
Description
Scalar fns are persisted as ScalarFnArray encodings via ScalarFnArrayPlugin (vortex-array/src/arrays/scalar_fn/plugin.rs). The encoding id is the fn's ScalarFnVTable::id(), and literal operands are ConstantArray children. Each fn's id string and options encoding therefore become permanent. This issue lists what to check for each fn.
Checks that apply to every fn:
- Is the id consistent with the other ids? See the naming table below.
- Do the fn's options reach the array metadata? Today the plugin never calls
ScalarFnVTable::serialize, and the tensor impls write only child dtypes. - Does
deserialize(serialize(o)) == ohold? - What do empty bytes decode to?
- Are unknown fields and stray bytes rejected or ignored?
- Is serializability explicit? The defaults at
vtable.rs:106-119silently returnOk(None).
Naming consistency
The current ids mix several styles:
| Style | Ids |
|---|---|
| flat, with the domain folded into the name | vortex.is_not_null, vortex.byte_length, vortex.variant_get, vortex.json_to_variant, vortex.replace_time_zone, vortex.strpos, vortex.lower |
| dotted domain | vortex.list.contains, vortex.list.sum, vortex.ext.storage, vortex.st.*, vortex.tensor.*, vortex.bloom_filter.sbbf.contains |
| version suffix | _v2 (vortex.sum_v2) vs .v2 (fastlanes.for.v2) |
| hyphen | vortex.piecewise-sequence |
Past renames left no alias, so ids from before them no longer decode:
vortex.geo.*→vortex.st.*(fed7038b)vortex.tensor.l2_denorm→normalized→l2_normalize(3d372165,803a3dc8)
ScalarFnPlugin and ScalarFnArrayPlugin have only id(). They have no separate read and write ids, unlike ArrayPlugin after ea94807b.
Stage 1
-
vortex.get_item- The
GetItemOpts.pathfield holds a single name. - The proto comment says
vortex.getitem(expr.proto:46).
- The
-
vortex.binary- One id covers 12 ops with three contracts: comparison, Kleene and/or, and checked arithmetic.
BinaryOp Eq = 0, so an empty payload decodes toEq.- Arithmetic has no overflow option.
-
vortex.betweenBetweenOptshas two bools.- Its own doc says it "will shortly be removed" (
between/mod.rs:203-204).
-
vortex.not: payload ignored. -
vortex.is_null: payload ignored. -
vortex.is_not_null: payload ignored. -
vortex.is_nan: not serializable (the RowFn default returnsNone). -
vortex.cast:CastOptsholds onlytarget. There is no safe/try option and no rounding option. -
vortex.list.contains:ListContainsOpts{sql_null_semantics}. An empty payload gives the old behaviour.
Stage 2
-
vortex.like- The
\escape is hard-wired. - The
case_insensitivebool doesn't name a folding.
- The
-
vortex.lower- Payload ignored.
- Flat id with no string domain.
-
vortex.pack- The
pathsfield holds names. nullabledeclares a nullable dtype that is never null.
- The
-
vortex.select: decoding skips the field-name uniqueness check. -
vortex.merge- Options are a raw byte (
0x00=RightMost,0x01=Error), not proto. - The Rust
DefaultisError, but byte 0 means RightMost.
- Options are a raw byte (
-
vortex.case_when:num_childrenduplicates the child count. -
vortex.zip- Payload ignored.
- The argument order
(if_true, if_false, mask)differs from Arrow'szip.
-
vortex.mask- Payload ignored.
- Polarity: true keeps the value.
-
vortex.fill_null: payload ignored. -
vortex.ext.storage- Payload ignored.
- Dotted, unlike the other core ids.
Stage 3
-
vortex.strpos- Payload ignored.
- Flat id with no string domain.
-
vortex.byte_length: payload ignored. -
vortex.list.length: payload ignored. -
vortex.list.sum- It shares
NumericalAggregateOpts.skip_nanswith the aggregates, and an empty payload decodes to include-NaNs, which is not the default. - The shared message is used by frozen
vortex.min/vortex.maxzone maps. - There is no overflow option.
- It shares
-
vortex.variant_getdtype=Noneanddtype=Some(Variant)mean the same thing.- There is no strict-cast option.
- The id is flat (
variant_get).
-
vortex.replace_time_zonenull_on_non_existentis a bool.- The ambiguity policy is a string child.
-
vortex.json_to_variant- The options carry a shredding spec, which is a physical hint.
- There is no on-error option.
Stage 4
-
vortex.tensor.l2_norm- The expression payload is ignored.
- Array metadata holds only
input_dtype. - There is no accumulation option.
-
vortex.tensor.inner_product- The expression payload is ignored.
- Array metadata holds only
lhs_dtypeandrhs_dtype. - There is no accumulation option.
-
vortex.tensor.cosine_similarity: same asinner_product, and there is no zero-vector option. -
vortex.tensor.l2_normalize- No expression serializer.
- Array metadata holds only
input_dtype. - There is no accumulation or zero-vector option.
-
vortex.st.area- Payload ignored.
- There is no metric or CRS option.
-
vortex.st.length- Payload ignored.
- There is no metric or CRS option.
-
vortex.st.distance- Payload ignored.
- There is no metric or CRS option.
-
vortex.st.contains- Payload ignored.
- There is no CRS option.
-
vortex.st.intersects- Payload ignored.
- There is no CRS option.
-
vortex.st.convex_hull: payload ignored. -
vortex.st.envelope: payload ignored. -
vortex.st.collect: payload ignored. -
vortex.st.make_line: payload ignored.
Stage 5
-
vortex.list.transform: not serializable, because its options are a lambda and lambdas have no wire form. -
vortex.row_encode: options are a rawu32count followed by(desc, nulls_first)pairs, not proto. -
vortex.row_size: same asrow_encode. - Foreign placeholder: its own
deserializeuses arity 0.
Written as ConstantArray, not as a ScalarFnArray
-
vortex.literal- Check that it never appears as a
ScalarFnArrayencoding. - Decimal width follows the in-memory value (TODO at
scalar/proto.rs:405).
- Check that it never appears as a
-
vortex.list.prepared_setListContains::reduceswaps a constant list for an arity-0PreparedSetnode (list_contains/mod.rs:225-243).- Check that it is never written.
Not serializable
-
vortex.dynamic: options hold a closure. -
vortex.dynamic_expression: runtime state. -
vortex.stat- Registered by default.
- Options are an
AggregateFnRef.
-
vortex.lower_bound- Internal.
- The name reads as "lower bound" but means bounds of
lower().
-
vortex.row_count: placeholder. -
vortex.row_idx: arity-0 placeholder that the scan replaces. -
vortex.bloom_filter.sbbf.contains: options are raw 8 bytes owned by the aggregate.
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Audit: making every scalar fn serializable with a permanent wire contract[/-][+]Scalar fn serialization: ids and options, checked per fn[/+]