Skip to content

Scalar fn serialization: ids and options, checked per fn #11396

Open

Description

@joseph-isaacs

Scalar fns are persisted as ScalarFnArray encodings via ScalarFnArrayPlugin (vortex-array/src/arrays/scalar_fn/plugin.rs). The encoding id is the fn's ScalarFnVTable::id(), and literal operands are ConstantArray children. Each fn's id string and options encoding therefore become permanent. This issue lists what to check for each fn.

Checks that apply to every fn:

  • Is the id consistent with the other ids? See the naming table below.
  • Do the fn's options reach the array metadata? Today the plugin never calls ScalarFnVTable::serialize, and the tensor impls write only child dtypes.
  • Does deserialize(serialize(o)) == o hold?
  • What do empty bytes decode to?
  • Are unknown fields and stray bytes rejected or ignored?
  • Is serializability explicit? The defaults at vtable.rs:106-119 silently return Ok(None).

Naming consistency

The current ids mix several styles:

Style Ids
flat, with the domain folded into the name vortex.is_not_null, vortex.byte_length, vortex.variant_get, vortex.json_to_variant, vortex.replace_time_zone, vortex.strpos, vortex.lower
dotted domain vortex.list.contains, vortex.list.sum, vortex.ext.storage, vortex.st.*, vortex.tensor.*, vortex.bloom_filter.sbbf.contains
version suffix _v2 (vortex.sum_v2) vs .v2 (fastlanes.for.v2)
hyphen vortex.piecewise-sequence

Past renames left no alias, so ids from before them no longer decode:

  • vortex.geo.* → vortex.st.* (fed7038b)
  • vortex.tensor.l2_denorm → normalized → l2_normalize (3d372165, 803a3dc8)

ScalarFnPlugin and ScalarFnArrayPlugin have only id(). They have no separate read and write ids, unlike ArrayPlugin after ea94807b.

Stage 1

  • vortex.get_item
    • The GetItemOpts.path field holds a single name.
    • The proto comment says vortex.getitem (expr.proto:46).
  • vortex.binary
    • One id covers 12 ops with three contracts: comparison, Kleene and/or, and checked arithmetic.
    • BinaryOp Eq = 0, so an empty payload decodes to Eq.
    • Arithmetic has no overflow option.
  • vortex.between
    • BetweenOpts has two bools.
    • Its own doc says it "will shortly be removed" (between/mod.rs:203-204).
  • vortex.not: payload ignored.
  • vortex.is_null: payload ignored.
  • vortex.is_not_null: payload ignored.
  • vortex.is_nan: not serializable (the RowFn default returns None).
  • vortex.cast: CastOpts holds only target. There is no safe/try option and no rounding option.
  • vortex.list.contains: ListContainsOpts{sql_null_semantics}. An empty payload gives the old behaviour.

Stage 2

  • vortex.like
    • The \ escape is hard-wired.
    • The case_insensitive bool doesn't name a folding.
  • vortex.lower
    • Payload ignored.
    • Flat id with no string domain.
  • vortex.pack
    • The paths field holds names.
    • nullable declares a nullable dtype that is never null.
  • vortex.select: decoding skips the field-name uniqueness check.
  • vortex.merge
    • Options are a raw byte (0x00=RightMost, 0x01=Error), not proto.
    • The Rust Default is Error, but byte 0 means RightMost.
  • vortex.case_when: num_children duplicates the child count.
  • vortex.zip
    • Payload ignored.
    • The argument order (if_true, if_false, mask) differs from Arrow's zip.
  • vortex.mask
    • Payload ignored.
    • Polarity: true keeps the value.
  • vortex.fill_null: payload ignored.
  • vortex.ext.storage
    • Payload ignored.
    • Dotted, unlike the other core ids.

Stage 3

  • vortex.strpos
    • Payload ignored.
    • Flat id with no string domain.
  • vortex.byte_length: payload ignored.
  • vortex.list.length: payload ignored.
  • vortex.list.sum
    • It shares NumericalAggregateOpts.skip_nans with the aggregates, and an empty payload decodes to include-NaNs, which is not the default.
    • The shared message is used by frozen vortex.min/vortex.max zone maps.
    • There is no overflow option.
  • vortex.variant_get
    • dtype=None and dtype=Some(Variant) mean the same thing.
    • There is no strict-cast option.
    • The id is flat (variant_get).
  • vortex.replace_time_zone
    • null_on_non_existent is a bool.
    • The ambiguity policy is a string child.
  • vortex.json_to_variant
    • The options carry a shredding spec, which is a physical hint.
    • There is no on-error option.

Stage 4

  • vortex.tensor.l2_norm
    • The expression payload is ignored.
    • Array metadata holds only input_dtype.
    • There is no accumulation option.
  • vortex.tensor.inner_product
    • The expression payload is ignored.
    • Array metadata holds only lhs_dtype and rhs_dtype.
    • There is no accumulation option.
  • vortex.tensor.cosine_similarity: same as inner_product, and there is no zero-vector option.
  • vortex.tensor.l2_normalize
    • No expression serializer.
    • Array metadata holds only input_dtype.
    • There is no accumulation or zero-vector option.
  • vortex.st.area
    • Payload ignored.
    • There is no metric or CRS option.
  • vortex.st.length
    • Payload ignored.
    • There is no metric or CRS option.
  • vortex.st.distance
    • Payload ignored.
    • There is no metric or CRS option.
  • vortex.st.contains
    • Payload ignored.
    • There is no CRS option.
  • vortex.st.intersects
    • Payload ignored.
    • There is no CRS option.
  • vortex.st.convex_hull: payload ignored.
  • vortex.st.envelope: payload ignored.
  • vortex.st.collect: payload ignored.
  • vortex.st.make_line: payload ignored.

Stage 5

  • vortex.list.transform: not serializable, because its options are a lambda and lambdas have no wire form.
  • vortex.row_encode: options are a raw u32 count followed by (desc, nulls_first) pairs, not proto.
  • vortex.row_size: same as row_encode.
  • Foreign placeholder: its own deserialize uses arity 0.

Written as ConstantArray, not as a ScalarFnArray

  • vortex.literal
    • Check that it never appears as a ScalarFnArray encoding.
    • Decimal width follows the in-memory value (TODO at scalar/proto.rs:405).
  • vortex.list.prepared_set
    • ListContains::reduce swaps a constant list for an arity-0 PreparedSet node (list_contains/mod.rs:225-243).
    • Check that it is never written.

Not serializable

  • vortex.dynamic: options hold a closure.
  • vortex.dynamic_expression: runtime state.
  • vortex.stat
    • Registered by default.
    • Options are an AggregateFnRef.
  • vortex.lower_bound
    • Internal.
    • The name reads as "lower bound" but means bounds of lower().
  • vortex.row_count: placeholder.
  • vortex.row_idx: arity-0 placeholder that the scan replaces.
  • vortex.bloom_filter.sbbf.contains: options are raw 8 bytes owned by the aggregate.

Activity

  1. changed the title [-]Audit: making every scalar fn serializable with a permanent wire contract[/-] [+]Scalar fn serialization: ids and options, checked per fn[/+] on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions